What's wrong in my playbook to stop journald from logging ?

/var/log/messages is not updated any more!

I can reproduce the issue with this simple isolated playbook :

- name: Reproduce journald hang
  hosts: test
  vars:
    - keys:
        - kp_XXX.pub
        - kp_YYY.pub
  tasks:
    - name: Verify journalctl before
      command: journalctl --verify

    - name: Add SSH keys 
      authorized_key: user=cloud key="{{ item }}"
      with_file: keys

    - name: Verify journalctl after
      command: journalctl --verify

The key XXX will make journald hang, but not the YYY one.
Reproduced on a CentOs 7.2 with ansible 1.9.4

File kp_XXX.pub:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCLd9k03Hvf3QVL8+dYd1KZY9p1ju/RkxHr+t6l6YbMcMfYcLHW6lsNIw2aLC7qpRopQPe/prQZkbXQBy8sYzNUcVtohPTD/V6wX7RXDCiVME9uUztY96Wust1Uc4Z28DhWyC55WFKhetGzfyxK+hMrtORnzdruo/bxHKmGu3rT5HYquB8SlPN/cSG/7itwy6QkXsqzmQUbEvaLPZNwU7qd9LiySFxsbhI2vJz+FiBS+CzkoTKOSZt60I0jRs4wIjXOZjQApcgddGa2ls3vq5HH39Xdr66+PnRU/rrRpaMTrcOTLPzzeWUQoF8VbkSiDXsI8ds+M842DKAT0DFVXnR kp_XXX

File kp_YYY.pub:

ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQBM9IuDxubRnbFh1e1dvFSKE91vrME5h/nQMsZo1Bmt8FXIQ7wJdNh+ANLYyQA7Q0tiXD1n97QQ9r89iwHFEUZVSXc7VM01AE27N45ybfLmLwtNm+kny6ncoPy7+MHcOQS9Ra56u6Bi6xXUc7vM4pL2iB/m0GnUSmECZZ5EVuOpMeJltf04/+PldQGOqxp9BzVF8XKEPlW5uc6UBesPCoHpR9lGA5UIIYq1sUDIGBy3T7FEXu8KhiHrtb1wuDGJU62SqR/fxgJDypjAtedm41TFcTZOMqTR29KYCKC4OjRaUTu7kf4rWq7/HWJViK2NLaeoy9xyG1BUUTqrGY6qRo85 kp_YYY

Although both keys are added, the third task fails and journald hangs:

TASK: [Verify journalctl after] ***********************************************
failed: [test] => {"changed": true, "cmd": ["journalctl", "--verify"], "delta": "0:00:00.008351", "end": "2016-03-31 12:49:58.669585", "rc": 1, "start": "2016-03-31 12:49:58.661234", "warnings": []}
stderr: 248668: invalid object
File corruption detected at /run/log/journal/cf9b563caf2bc11cab56d6a504ff6a29/system.journal:248668 (of 8388608 bytes, 28%).
FAIL: /run/log/journal/cf9b563caf2bc11cab56d6a504ff6a29/system.journal (Cannot assign requested address)

dmesg logs plenty of thes lines:

[  761.806277] systemd-journald[366]: Failed to write entry (27 items, 719 bytes), ignoring: Cannot assign requested address
[  761.807514] systemd-journald[366]: Failed to write entry (23 items, 633 bytes), ignoring: Cannot assign requested address
[  761.859245] systemd-journald[366]: Failed to write entry (24 items, 718 bytes), ignoring: Cannot assign requested address

When verifying journald files with journalctl --verify:

248668: invalid object
File corruption detected at /run/log/journal/cf9b563caf2bc11cab56d6a504ff6a29/system.journal:248668 (of 8388608 bytes, 28%).
FAIL: /run/log/journal/cf9b563caf2bc11cab56d6a504ff6a29/system.journal (Cannot assign requested address)

Is this an error from me, from ansible, or centos?

How can it be fixed?

Found these 2 related links:
https://access.redhat.com/solutions/2117311?tour=6#comments
Claudio's blog

  • Did you try setting up the keys manually on the host and then run journalctl to see if you could reproduce the issue? – Petro026 Mar 31 '16 at 15:07
  • Yes, setting the keys manually is not an issue. When running the playbook again, even if the task is OK, the journal file gets corrupted. – Gwenn Apr 1 '16 at 15:49
  • open the console!
  • get to be root
  • examine the output of "journalctl -xn" and "dmesg"
  • if you get this or something similar

    [ 12.713167] systemd-journald[113]: Failed to write entry (9 items, 245 bytes), ignoring: Cannot assign requested address

  • verify the journals with "journalctl --verify"

  • check if the logs are within sizes "journalctl --disk-usage"
  • by then you have an idea of the corrupted logs - they show up iin red, no way you can ignore them
  • create a new folder move these files.
  • do a journalctl --verify re run of the verify and to make sure
  • reboot

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.