Sparks on Yarn How to View Task Log

Both Flink and Spark support self-built clusters. However, in order to ensure stability and resource isolation, it is best to run tasks in a production environment with the help of a resource management framework (such as Yarn). When the task is running on yarn, it may be inconvenient to query the log, especially after the task process exits abnormally.

After the yarn container exits, the log will not be saved by default. So you need to start JobHistoryServer, there are many tutorials on the internet.

Running the Spark task can be viewed directly through the spark web ui:

For the yarn application that has ended, the spark process has exited, and the webui service cannot be provided.

According to the way Flink On Yarn looks at the task log, we already know the url composition of the log. This time, just find the container name and node to access the log.

The address of the driver is: http://node5:19888/jobhistory/logs/node3: 8041/container _1634207619484 _ 0496 _ 0/kloc. container _ 16342076 19484 _ 0496 _ 0 1 _ 0000 1/root/stderr/？ Start =0

Search the driver log and find the container name container _1634207619484 _ 0496 _ 01_ 000002 and host node 3.

It is very easy to check the running logs of flink/spark, because they all provide web ui services. However, when the task exits abnormally, the flink/spark process ends and web ui service cannot be provided. We use the job history server to save and display the logs at that time. However, yarn's web only shows the log link of flink job manager /spark driver, and we need to splice the log link of flink task manager /spark executor ourselves.

Finally, I have a question: the composition of the URL introduced in this article is speculative, in which the third part/container _163420719484 _ 0505 _ 01_ 000001container _1. I hope that friends who know can leave a message to dispel doubts.