I did the lab using the console first and actually got the expected result first time, or so I thought. I eventually realized that things weren’t as they seemed. While the file gets created and copied to the bucket, I couldn’t see any events related to the startup script in the StackDriver logs.
By looking at syslog, I also noticed that stress wasn’t running either. In fact, all of the failures seem to be linked to this error message : "dpkg was interrupted, you must manually run ‘dpkg –configure -a’ to correct the problem."
If you look at the extract below, this seems to impact both the installation of the StackDriver agent (which explains why I can’t see anything in the logs) and of the stress utility. Looking at the lab walkthrough afterwards, I can’t see what I am doing differently that would cause this behaviour. I played with various scopes, re-created a few projects/VMs to no avail. Any pointers?
I haven’t tried running the ‘dpkg –configure -a’ command yet, as I haven’t got access to the shell from work. However, even if it fixes broken links or whatever else that command does, I shouldn’t have to run it in the first place, should I?
Sep 9 02:17:23 mygcechallengevm startup-script: INFO startup-script: + apt-get upgrade -y
Sep 9 02:17:24 mygcechallengevm startup-script: INFO startup-script: E: dpkg was interrupted, you must manually run ‘dpkg –configure -a’ to correct the problem.
Sep 9 02:17:24 mygcechallengevm startup-script: INFO startup-script: + curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
Sep 9 02:17:25 mygcechallengevm startup-script: INFO startup-script: + bash install-logging-agent.sh
Sep 9 02:17:25 mygcechallengevm startup-script: INFO startup-script: ==============================================================================
Sep 9 02:17:25 mygcechallengevm startup-script: INFO startup-script: Starting installation of google-fluentd
Sep 9 02:17:25 mygcechallengevm startup-script: INFO startup-script: ==============================================================================
Sep 9 02:17:25 mygcechallengevm startup-script: INFO startup-script: Installing agents for Debian or Ubuntu.
Sep 9 02:17:25 mygcechallengevm startup-script: INFO startup-script: Warning: apt-key output should not be parsed (stdout is not a terminal)
Sep 9 02:17:25 mygcechallengevm startup-script: INFO startup-script: OK
Sep 9 02:17:25 mygcechallengevm startup-script: INFO startup-script: E: dpkg was interrupted, you must manually run ‘dpkg –configure -a’ to correct the problem.
Sep 9 02:17:26 mygcechallengevm startup-script: INFO startup-script: dpkg-query: no packages found matching google-fluentd
Sep 9 02:17:26 mygcechallengevm startup-script: INFO startup-script: Failed to restart google-fluentd.service: Unit google-fluentd.service not found.
Sep 9 02:17:26 mygcechallengevm root: Test syslog message from google-fluentd- at Mon Sep 9 02:17:26 UTC 2019.
Sep 9 02:17:26 mygcechallengevm startup-script: INFO startup-script: ==============================================================================
Sep 9 02:17:26 mygcechallengevm startup-script: INFO startup-script: Errors occurred while installing google-fluentd-. See the log snippet
Sep 9 02:17:26 mygcechallengevm startup-script: INFO startup-script: above or run:
Sep 9 02:17:26 mygcechallengevm startup-script: INFO startup-script: sudo service google-fluentd status
Sep 9 02:17:26 mygcechallengevm startup-script: INFO startup-script: ==============================================================================
Sep 9 02:17:26 mygcechallengevm startup-script: INFO startup-script: + apt-get install stress
Sep 9 02:17:26 mygcechallengevm startup-script: INFO startup-script: E: dpkg was interrupted, you must manually run ‘dpkg –configure -a’ to correct the problem.
Sep 9 02:17:26 mygcechallengevm startup-script: INFO startup-script: + stress -c 8 -t 120
Sep 9 02:17:26 mygcechallengevm startup-script: INFO startup-script: /startup-d7qx5wen/tmpz4r_jvoa: line 28: stress: command not found
3 Answers
FWIW, running the dpkg command eventually fixed the problem – I now can see all the logs in StackDriver as well as the stress CPU spike.
Hey, thanks so much for reporting this, Christophe! I have now updated the startup script to hopefully avoid this for everyone. Please let me know if you have any more troubles! And thanks, again!
Mattias
No worries. FYI, I had another issue with the script failing/stopping because of clock synchronization issues. Switching to Debian 10, rather than the default of Debian 9, appeared to solve that issue.
Glad you worked around that! I’ll respond more fully in an "answer", so you get a notification.
Oi! I have seen a message of kernel reports TIME_ERROR: 0x41: Clock Unsynchronized–but in my case, this came well after the script had completed successfully and thus seemed quite ignorable. Hmmm. Well, I am definitely glad that you were able to work around the issues, anyway, and I hope that if anyone else does hit this then this thread can at least offer some insight.
I hope on the whole you’re enjoying the course and learning lots!
Mattias
Running the dpkg command eventually fixed the problem – I now can see all the logs in StackDriver as well as the stress CPU spike.