Kubernetes 1.27: Query Node Logs Using The Kubelet API
Author: Aravindh Puthiyaparambil (Red Hat)
Kubernetes 1.27 introduced a new feature called Node log query that allows viewing logs of services running on the node.
What problem does it solve?
Cluster administrators face issues when debugging malfunctioning services running on the node. They usually have to SSH or RDP into the node to view the logs of the service to debug the issue. The Node log query feature helps with this scenario by allowing the cluster administrator to view the logs using kubectl. This is especially useful with Windows nodes where you run into the issue of the node going to the ready state but containers not coming up due to CNI misconfigurations and other issues that are not easily identifiable by looking at the Pod status.
How does it work?
The kubelet already has a /var/log/ viewer that is accessible via the node
proxy endpoint. The feature supplements this endpoint with a shim that shells
out to journalctl, on Linux nodes, and the Get-WinEvent cmdlet on Windows
nodes. It then uses the existing filters provided by the commands to allow
filtering the logs. The kubelet also uses heuristics to retrieve the logs.
If the user is not aware if a given system services logs to a file or to the
native system logger, the heuristics first checks the native operating system
logger and if that is not available it attempts to retrieve the first logs
from /var/log/<servicename> or /var/log/<servicename>.log or
/var/log/<servicename>/<servicename>.log.
On Linux we assume that service logs are available via journald, and that
journalctl is installed. On Windows we assume that service logs are available
in the application log provider. Also note that fetching node logs is only
available if you are authorized to do so (in RBAC, that's get and
create access to nodes/proxy). The privileges that you need to fetch node
logs also allow elevation-of-privilege attacks, so be careful about how you
manage them.
How do I use it?
To use the feature, ensure that the NodeLogQuery
feature gate is
enabled for that node, and that the kubelet configuration options
enableSystemLogHandler and enableSystemLogQuery are both set to true. You can
then query the logs from all your nodes or just a subset. Here is an example to
retrieve the kubelet service logs from a node:
# Fetch kubelet logs from a node named node-1.example
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet"
You can further filter the query to narrow down the results:
# Fetch kubelet logs from a node named node-1.example that have the word "error"
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet&pattern=error"
You can also fetch files from /var/log/ on a Linux node:
kubectl get --raw "/api/v1/nodes/<insert-node-name-here>/proxy/logs/?query=/<insert-log-file-name-here>"
You can read the documentation for all the available options.
How do I help?
Please use the feature and provide feedback by opening GitHub issues or reaching out to us on the #sig-windows channel on the Kubernetes Slack or the SIG Windows mailing list.