QtCS2017 Discuss Qt Logging enhancements

Discussion

• Structured logging:
  • We want to extract information from logs and detect common messages. For example:

From	To
Got message "Hello"	Got message "%s" Hello
Got message "World"	Got message "%s" World

• • We want to allow logging to store ancillary data, possibly in machine-readable format, likely not part of the message itself
    • Like journald API can do
    • How do we allow different formatting depending on backend? Needs to support formatting user types too.
• Structured output
  • Store to databases, for example
  • Do we still obey QT_MESSAGE_PATTERN?
  • Select backend with environment variable

Need to review with LTTNG and ETW tracing

Executive Summary

Qt Logging lacks regarding 2 different use cases:

1/ High-performance code (say HW handling, big data processing) needs a way to inject a few structured tracepoints to generate large binary traces (say, for profiling)

2/ Existing tracepoints (think about nb of qDebug in Qt user code) should generate more structured traces to:

• facilitate exploration and diagnostic, and
• allow immediate use of analysis tools like Python

Service and Device developers need #1. Medical Devices would benefit from #2 for post-market surveillance. Device designers operating in an uncontrolled environment would probably benefit from #2 as well

Prototypes

• LTTNG and ETW tracing uses external tracepoint data description to generate code for LTTNG and ETW backends (or no code at all)
• Qt-based modmedLog library (see enclosed presentation) uses Bind<T> and IData implementations to bind tracepoint data to JSON/XML/Console outputs and possibly read them back for replay

Conclusion

We should sit down and see how these 2 use cases intersect and the best things that are provided by each prototype

Requirements

For #1:

• as little generated code as possible at the tracepoint:
  • no binary or numeric formatting
  • completely removable from release code (not just bypassing like QNoDebug::operator<<)

For #2:

• source compatibility with qDebug tracepoints
  • do not require non trivial changes such as changing from qDebug()<< to qDebug(...)
  • allow changing qDebug() to qTrace() or whatever to provide choice of old vs structured tracing
  • extensible to any user data type
• offer structured trace formats (TSV+JSON, XML, sqlite?) to avoid low-level parsing

Open Questions

• Can we generate efficient tracepoint code without an external tracepoint description and tool?
• Should we provide a format+args handler delegating by default to the existing handler?

(*QtMessageHandler2)(...,const QString & format, const QStringList & args);

• Should we provide a formatting-independant handler based on Bind<T>:

(*QtMessageHandler3)(...,const QString & format, const QList<BindableRef> & args);

• What about modmedLog performance?
• When variadic macros and templates are available, can they be used to get the best of both prototypes?