YARN log aggregation

Log aggregation is enabled/disabled through the configuration parameter yarn.logaggregation.enable. By default, it is disabled.

When log aggregation is enabled, it copies container log files into a Hadoop filesystem (most probably HDFS) after a application has finished its job. The location is specified by $yarn.nodemanager.remote-app-log-dir/${user}/${yarn.nodemanager.remote-app-log-dir-suffix}/<application-id> ( see yarn.nodemanager.remote-app-log-dir, yarn.nodemanager.remote-app-log-dir-suffix, application id )

After copying the log files, the local log files are retained for yarn.nodemanager.delete.debug-delay-sec seconds (possibly for 0 seconds).

When log aggregation is enabled, the parameter yarn.log.server.url (set in yarn-site.xml) should point at the job history server in order for the resource manager UI to be able to display logs.