Skip to content

k3log

Action-CI Documentation Status Package

A collection of log utilities for Python logging. Provides helpers for creating loggers, formatters, and file handlers with sensible defaults.

k3log is a component of pykit3 project: a python3 toolkit set.

Installation

pip install k3log

Quick Start

import k3log

# Create a logger with default settings
logger = k3log.make_logger("/tmp/myapp")
logger.info("Hello World")

# Add stdout handler to root logger
k3log.add_std_handler(logger)

# Create file handler
handler = k3log.make_file_handler("/tmp/myapp.log")
logger.addHandler(handler)

# Get stack trace as formatted string
stack = k3log.stack_str()
logger.debug("Stack trace: %s", stack)

API Reference

k3log

add_std_handler(logger, stream=None, fmt=None, datefmt=None, level=None)

It adds a stdout or stderr steam handler to the logger.

Parameters:

Name Type Description Default
logger Logger

is an instance of logging.Logger to add handler to.

required
stream TextIO | str | None

specifies the stream, it could be: - sys.stdout or a string stdout. - sys.stderr or a string stderr.

None
fmt str

is the log message format. It can be an alias name(like default) that can be used in get_fmt(). By default it is default: [%(asctime)s,%(process)d-%(thread)d,%(filename)s,%(lineno)d,%(levelname)s] %(message)s.

None
datefmt str

is the format for date. It can be an alias name(like time) that can be used in get_datefmt(). By default it is None.

None
level int | str | None

is the log level. It can be int value such as logging.DEBUG or string such as DEBUG. By default it is the logger's level.

None

Returns:

Type Description
Logger

the logger in argument.

Source code in k3log/log.py
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
def add_std_handler(logger, stream=None, fmt=None, datefmt=None, level=None):
    """
    It adds a `stdout` or `stderr` steam handler to the `logger`.

    Args:
        logger(logging.Logger):
            is an instance of `logging.Logger` to add handler to.

        stream(typing.TextIO | str | None):
            specifies the stream, it could be:
            -   ``sys.stdout`` or a string ``stdout``.
            -   ``sys.stderr`` or a string ``stderr``.

        fmt(str):
            is the log message format.
            It can be an alias name(like `default`) that can be used in
            ``get_fmt()``.
            By default it is ``default``:
            ``[%(asctime)s,%(process)d-%(thread)d,%(filename)s,%(lineno)d,%(levelname)s] %(message)s``.

        datefmt(str):
            is the format for date.
            It can be an alias name(like `time`) that can be used in
            `get_datefmt()`.
            By default it is `None`.

        level(int | str | None):
            is the log level.
            It can be int value such as ``logging.DEBUG`` or string such as ``DEBUG``.
            By default it is the logger's level.

    Returns:
        (logging.Logger): the `logger` in argument.
    """

    stream = stream or sys.stdout

    if stream == "stdout":
        stream = sys.stdout

    elif stream == "stderr":
        stream = sys.stderr

    stdhandler = logging.StreamHandler(stream)
    stdhandler.setFormatter(make_formatter(fmt=fmt, datefmt=datefmt))
    if level is not None:
        stdhandler.setLevel(level)

    logger.addHandler(stdhandler)

    return logger

deprecate(msg=None, fmt=None, sep=None)

Print a deprecate message, at warning level. The printed message includes:

  • User defined message msg,
  • And calling stack of where this warning occurs. <frame-n> is where deprecate is called.

::

<msg> Deprecated: <frame-1> --- <frame-2> --- ... --- <frame-n>

The default frame format is {fn}:{ln} in {func} {statement}. It can be changed with argument fmt. Frame separator by default is ---, and can be changed with argument sep.

For example, the following statement::

def foo():
    deprecate('should not be here.',
              fmt="{fn}:{ln} in {func}\n  {statement}",
              sep="\n"
              )

Would produce a message::

Deprecated: should not be here.
runpy.py:174 in _run_module_as_main
  "__main__", fname, loader, pkg_name)
runpy.py:72 in _run_code
  exec code in run_globals
...
test_log.py:82 in test_deprecate
  deprecate()
  'foo', fmt='{fn}:{ln} in {func}\n  {statement}', sep='\n')

Parameters:

Name Type Description Default
msg str | None

is description of the deprecated statement. It could be None.

None
fmt str | None

is call stack frame format. By default it is {fn}:{ln} in {func} {statement}.

None
sep str | None

is the separator string between each frame. By default it is " --- ". Thus all frames are printed in a single line.

None
Source code in k3log/log.py
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
def deprecate(msg=None, fmt=None, sep=None):
    """
    Print a `deprecate` message, at warning level.
    The printed message includes:

    -   User defined message `msg`,
    -   And calling stack of where this warning occurs.
        `<frame-n>` is where `deprecate` is called.

    ::

        <msg> Deprecated: <frame-1> --- <frame-2> --- ... --- <frame-n>

    The default frame format is `{fn}:{ln} in {func} {statement}`.
    It can be changed with argument `fmt`.
    Frame separator by default is ` --- `, and can be changed with argument `sep`.

    For example, the following statement::

        def foo():
            deprecate('should not be here.',
                      fmt="{fn}:{ln} in {func}\\n  {statement}",
                      sep="\\n"
                      )

    Would produce a message::

        Deprecated: should not be here.
        runpy.py:174 in _run_module_as_main
          "__main__", fname, loader, pkg_name)
        runpy.py:72 in _run_code
          exec code in run_globals
        ...
        test_log.py:82 in test_deprecate
          deprecate()
          'foo', fmt='{fn}:{ln} in {func}\\n  {statement}', sep='\\n')

    Args:
        msg(str | None):
            is description of the `deprecated` statement.
            It could be `None`.

        fmt(str | None):
            is call stack frame format.
            By default it is `{fn}:{ln} in {func} {statement}`.

        sep(str | None):
            is the separator string between each frame.
            By default it is ``" --- "``.
            Thus all frames are printed in a single line.
    """

    d = "Deprecated:"

    if msg is not None:
        d += " " + str(msg)

    logger.warning(d + (sep or default_stack_sep) + stack_str(offset=1, fmt=fmt, sep=sep))

get_datefmt(datefmt)

It translates a predefined datefmt alias to actual datefmt. Predefined alias includes::

{
    'default':  None,       # use the system default datefmt
    'time':     '%H:%M:%S',
}

Parameters:

Name Type Description Default
datefmt str | None

is the alias name. If no predefined alias name is found, it returns the passed in value of datefmt.

required

Returns:

Type Description
str | None

translated datefmt or the original value of argument datefmt.

Source code in k3log/log.py
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
def get_datefmt(datefmt):
    """
    It translates a predefined datefmt alias to actual datefmt.
    Predefined alias includes::

        {
            'default':  None,       # use the system default datefmt
            'time':     '%H:%M:%S',
        }

    Args:
        datefmt(str | None):
            is the alias name.
            If no predefined alias name is found, it returns the passed in value of
            `datefmt`.

    Returns:
        (str | None): translated `datefmt` or the original value of argument `datefmt`.
    """

    if datefmt is None:
        return datefmt

    return date_formats.get(datefmt, datefmt)

get_fmt(fmt)

It translate a predefined fmt alias to actual fmt. Predefined alias includes::

{
    'default':    '[%(asctime)s,%(process)d-%(thread)d,%(filename)s,%(lineno)d,%(levelname)s] %(message)s',
    'time_level': "[%(asctime)s,%(levelname)s] %(message)s",
    'message':    '%(message)s',
}

Parameters:

Name Type Description Default
fmt str | None

is the alias name. If no predefined alias name is found, it returns the passed in value of fmt.

required

Returns:

Type Description
str

translated fmt or the original value of argument fmt.

Source code in k3log/log.py
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
def get_fmt(fmt):
    """
    It translate a predefined fmt alias to actual fmt.
    Predefined alias includes::

        {
            'default':    '[%(asctime)s,%(process)d-%(thread)d,%(filename)s,%(lineno)d,%(levelname)s] %(message)s',
            'time_level': "[%(asctime)s,%(levelname)s] %(message)s",
            'message':    '%(message)s',
        }

    Args:
        fmt(str | None):
            is the alias name.
            If no predefined alias name is found, it returns the passed in value of
            `fmt`.

    Returns:
        (str): translated `fmt` or the original value of argument `fmt`.
    """

    if fmt is None:
        fmt = "default"

    return log_formats.get(fmt, fmt)

get_root_log_fn()

Returns the log file name for root logger. The log file name suffix is .out.

  • pyfn.out: if program is started with python pyfn.py.
  • __instant_command__.out: if instance python command is called: python -c "import k3log; print k3log.get_root_log_fn().
  • __stdin__.out: if python statement is passed in as stdin: echo "from pykit import k3log; print k3log.get_root_log_fn()" | python.

Returns:

Type Description
str

log file name.

Source code in k3log/log.py
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
def get_root_log_fn():
    """
    Returns the log file name for root logger.
    The log file name suffix is ``.out``.

    -   ``pyfn.out``: if program is started with ``python pyfn.py``.
    -   ``__instant_command__.out``: if instance python command is called:
        ``python -c "import k3log; print k3log.get_root_log_fn()``.
    -   ``__stdin__.out``: if python statement is passed in as stdin:
        ``echo "from pykit import k3log; print k3log.get_root_log_fn()" | python``.

    Returns:
        (str): log file name.
    """

    if hasattr(__main__, "__file__"):
        name = __main__.__file__
        name = os.path.basename(name)
        if name == "<stdin>":
            name = "__stdin__"
        return name.rsplit(".", 1)[0] + "." + log_suffix
    else:
        return "__instant_command__." + log_suffix

make_file_handler(base_dir=None, log_fn=None, fmt=None, datefmt=None, tag=None)

It creates a rolling log file handler.

A rolling file can be removed at any time. This handler checks file status everytime write log to it. If file is changed(removed or moved), this handler automatically creates a new log file.

Parameters:

Name Type Description Default
base_dir str | None

specifies the dir of log file. If it is None, use k3conf.log_dir as default.

None
log_fn str | None

specifies the log file name. If it is None, use k3log.get_root_log_fn to make a log file name.

None
fmt str | None

specifies log format. It can be an alias that can be used in k3log.get_fmt(), or None to used the default log format.

None
datefmt str | None

specifies log date format. It can be an alias that can be used in k3log.get_datefmt(), or None to used the default date format.

None

Returns:

Type Description
FileHandler

a handler that reopens the log file after log rotation moves it.

Source code in k3log/log.py
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
def make_file_handler(base_dir=None, log_fn=None, fmt=None, datefmt=None, tag=None):
    """
    It creates a rolling log file handler.

    A rolling file can be removed at any time.
    This handler checks file status everytime write log to it.
    If file is changed(removed or moved), this handler automatically creates a new
    log file.

    Args:
        base_dir(str | None):
            specifies the dir of log file.
            If it is ``None``, use ``k3conf.log_dir`` as default.

        log_fn(str | None):
            specifies the log file name.
            If it is ``None``, use ``k3log.get_root_log_fn`` to make a log file name.

        fmt(str | None):
            specifies log format.
            It can be an alias that can be used in ``k3log.get_fmt()``, or ``None`` to
            used the ``default`` log format.

        datefmt(str | None):
            specifies log date format.
            It can be an alias that can be used in ``k3log.get_datefmt()``, or ``None`` to
            used the ``default`` date format.

    Returns:
        (logging.FileHandler): a handler that reopens the log file after log rotation moves it.

    """
    if base_dir is None:
        base_dir = k3confloader.conf.log_dir
    if log_fn is None:
        log_fn = get_root_log_fn()
    file_path = os.path.join(base_dir, log_fn)

    handler = FixedWatchedFileHandler(file_path, tag=tag)
    handler.setFormatter(make_formatter(fmt=fmt, datefmt=datefmt))

    return handler

make_formatter(fmt=None, datefmt=None)

It creates an logging.Formatter instance, with specified fmt and datefmt.

Parameters:

Name Type Description Default
fmt str | None

specifies log format. It can be an alias that can be used in get_fmt(), or None to used the default log format.

None
datefmt str | None

specifies log date format. It can be an alias that can be used in get_datefmt(), or None to used the default date format.

None

Returns:

Type Description
Formatter

the formatter.

Source code in k3log/log.py
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
def make_formatter(fmt=None, datefmt=None):
    """
    It creates an `logging.Formatter` instance, with specified `fmt` and `datefmt`.

    Args:
        fmt(str | None):
            specifies log format.
            It can be an alias that can be used in `get_fmt()`, or `None` to
            used the `default` log format.

        datefmt(str | None):
            specifies log date format.
            It can be an alias that can be used in `get_datefmt()`, or `None` to
            used the `default` date format.

    Returns:
        (logging.Formatter): the formatter.
    """

    fmt = get_fmt(fmt)
    datefmt = get_datefmt(datefmt)

    return logging.Formatter(fmt=fmt, datefmt=datefmt)

make_logger(base_dir=None, log_name=None, log_fn=None, level=logging.DEBUG, fmt=None, datefmt=None)

It creates a logger with a rolling file hander and specified formats.

Parameters:

Name Type Description Default
base_dir str | None

specifies the dir of log file. If it is None, use config.log_dir as default.

None
log_name str | None

is the name of the logger to create. None means the root logger.

None
log_fn str | None

specifies the log file name. If it is None, use k3log.get_root_log_fn to make a log file name.

None
level int | str

specifies log level. It could be int value such as logging.DEBUG or string such as DEBUG.

DEBUG
fmt str | None

specifies log format. It can be an alias that can be used in k3log.get_fmt(), or None to used the default log format.

None
datefmt str | None

specifies log date format. It can be an alias that can be used in k3log.get_datefmt(), or None to used the default date format.

None

Returns:

Type Description
Logger

the configured logger.

Source code in k3log/log.py
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
def make_logger(base_dir=None, log_name=None, log_fn=None, level=logging.DEBUG, fmt=None, datefmt=None):
    """
    It creates a logger with a rolling file hander and specified formats.

    Args:
        base_dir(str | None):
            specifies the dir of log file.
            If it is ``None``, use ``config.log_dir`` as default.

        log_name(str | None):
            is the name of the logger to create.
            ``None`` means the root logger.

        log_fn(str | None):
            specifies the log file name.
            If it is ``None``, use ``k3log.get_root_log_fn`` to make a log file name.

        level(int | str):
            specifies log level.
            It could be int value such as ``logging.DEBUG`` or string such as ``DEBUG``.

        fmt(str | None):
            specifies log format.
            It can be an alias that can be used in ``k3log.get_fmt()``, or ``None`` to
            used the ``default`` log format.

        datefmt(str | None):
            specifies log date format.
            It can be an alias that can be used in ``k3log.get_datefmt()``, or ``None`` to
            used the `default` date format.

    Returns:
        (logging.Logger): the configured logger.

    """

    # if log_name is None, get the root logger
    logger = logging.getLogger(log_name)
    logger.setLevel(level)

    if base_dir is None:
        base_dir = k3confloader.conf.log_dir

    if log_fn is None:
        if log_name is None:
            log_fn = get_root_log_fn()
        else:
            log_fn = log_name + "." + log_suffix

    #  # do not add 2 handlers to one logger by default
    for h in list(logger.handlers):
        if getattr(h, "tag", None) == "root":
            logger.removeHandler(h)
            h.close()

    logger.addHandler(make_file_handler(base_dir, log_fn, fmt=fmt, datefmt=datefmt, tag="root"))

    return logger

set_logger_level(level=logging.INFO, name_prefixes=None)

Set all logger level that matches name_prefixes.

Parameters:

Name Type Description Default
level int | str

specifies log level. It could be int value such as logging.DEBUG or string such as DEBUG.

INFO
name_prefixes str | tuple[str, ...] | None

specifies log prefixes which is operated. It can be None, str or a tuple of str. If name_prefixes is None, set the log level for all logger.

None
Source code in k3log/log.py
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
def set_logger_level(level=logging.INFO, name_prefixes=None):
    """
    Set all logger level that matches ``name_prefixes``.

    Args:
        level(int | str):
            specifies log level.
            It could be int value such as ``logging.DEBUG`` or string such as ``DEBUG``.

        name_prefixes(str | tuple[str, ...] | None):
            specifies log prefixes which is operated.
            It can be None, str or a tuple of str.
            If `name_prefixes` is None, set the log level for all logger.
    """

    if name_prefixes is None:
        name_prefixes = ("",)

    root_logger = logging.getLogger()

    loggers = sorted(root_logger.manager.loggerDict.items())
    # loggerDict does not hold the root logger, and getLogger("root") returns it.
    loggers.append((root_logger.name, root_logger))

    for name, _ in loggers:
        if name.startswith(name_prefixes):
            name_logger = logging.getLogger(name)
            name_logger.setLevel(level)

stack_format(stacks, fmt=None, sep=None)

It formats stack made from k3log.stack_list.

With fmt="{fn}:{ln} in {func}\n {statement}" and sep="\n", a formatted stack would be::

runpy.py:174 in _run_module_as_main
  "__main__", fname, loader, pkg_name)
runpy.py:72 in _run_code
  exec code in run_globals
...
test_logutil.py:82 in test_deprecate
  k3log.deprecate()
  'foo', fmt='{fn}:{ln} in {func}\n  {statement}', sep='\n')

Parameters:

Name Type Description Default
stacks list[FrameSummary]

is stack from k3log.stack_list. Each frame is read as (filename, line number, function name, statement).

required
fmt str | None

specifies the template to format a stack frame. It can use {fn} (the base name of the file), {ln}, {func} and {statement}. By default it is: {fn}:{ln} in {func} {statement}.

None
sep str | None

specifies the separator string between each stack frame. By default it is " --- ". Thus all frames are in the same line.

None

Returns:

Type Description
str

a string representing a calling stack.

Source code in k3log/log.py
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
def stack_format(stacks, fmt=None, sep=None):
    """
    It formats stack made from ``k3log.stack_list``.

    With ``fmt="{fn}:{ln} in {func}\\n  {statement}"``
    and ``sep="\\n"``, a formatted stack would be::

        runpy.py:174 in _run_module_as_main
          "__main__", fname, loader, pkg_name)
        runpy.py:72 in _run_code
          exec code in run_globals
        ...
        test_logutil.py:82 in test_deprecate
          k3log.deprecate()
          'foo', fmt='{fn}:{ln} in {func}\\n  {statement}', sep='\\n')

    Args:
        stacks(list[traceback.FrameSummary]):
            is stack from ``k3log.stack_list``.
            Each frame is read as ``(filename, line number, function name, statement)``.

        fmt(str | None):
            specifies the template to format a stack frame.
            It can use ``{fn}`` (the base name of the file), ``{ln}``, ``{func}`` and ``{statement}``.
            By default it is: ``{fn}:{ln} in {func} {statement}``.

        sep(str | None):
            specifies the separator string between each stack frame.
            By default it is ``" --- "``.
            Thus all frames are in the same line.

    Returns:
        (str): a string representing a calling stack.
    """

    if fmt is None:
        fmt = "{fn}:{ln} in {func} {statement}"

    if sep is None:
        sep = default_stack_sep

    dict_stacks = []
    for st in stacks:
        o = {
            "fn": os.path.basename(st[0]),
            "ln": st[1],
            "func": st[2],
            "statement": st[3],
        }
        dict_stacks.append(o)

    return sep.join([fmt.format(**xx) for xx in dict_stacks])

stack_list(offset=0)

It returns the calling stack from where it is called.

Parameters:

Name Type Description Default
offset int

remove the lowest offset frames.

0

Returns:

Type Description
list

traceback.FrameSummary frames, the outermost first. A frame indexes as (filename, line number, function name, statement).

Source code in k3log/log.py
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
def stack_list(offset=0):
    """
    It returns the calling stack from where it is called.

    Args:
        offset(int):
            remove the lowest `offset` frames.

    Returns:
        (list): `traceback.FrameSummary` frames, the outermost first. A frame indexes as
            `(filename, line number, function name, statement)`.
    """

    offset += 1  # count this function as 1

    # list of ( filename, line-nr, in-what-function, statement )
    x = traceback.extract_stack()
    return x[:-offset]

stack_str(offset=0, fmt=None, sep=None)

It creates a string representing calling stack from where this function is called.

Parameters:

Name Type Description Default
offset int

remove the lowest offset frames. Because usually one does not need the frame of the k3log.stack_str line.

0
fmt str | None

is same as k3log.stack_format.

None
sep str | None

is same as k3log.stack_format.

None

Returns:

Type Description
str

a string representing a calling stack.

Source code in k3log/log.py
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
def stack_str(offset=0, fmt=None, sep=None):
    """
    It creates a string representing calling stack from where this function is
    called.

    Args:
        offset(int):
            remove the lowest `offset` frames.
            Because usually one does not need the frame of the `k3log.stack_str`
            line.

        fmt(str | None): is same as `k3log.stack_format`.

        sep(str | None): is same as `k3log.stack_format`.

    Returns:
        (str): a string representing a calling stack.
    """
    offset += 1  # count this function as 1
    return stack_format(stack_list(offset), fmt=fmt, sep=sep)

License

The MIT License (MIT) - Copyright (c) 2015 Zhang Yanpo (张炎泼)