[AIRFLOW-1422] Customize File Handler for logging module via airflow.cfg

[ronfung]

Dear Airflow maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-1422] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-1422

Description

• # Allow log files generated by the scheduler, webserver and workers be more organized based on time interval, size etc. Make it easier for analysis, maintenance and cleanup.
• # Log Handler - it supports all file handlers: TimedRotatingFileHandler, RotatingFileHandler, WatchedFileHandler, FileHandler or StreamHandler.
• #Two configuration keys are added: log_handler_class and log_handler_params. Default is using logging.FileHandler with None.
  e.g. 1
  log_handler_class = logging.handlers.RotatingFileHandler
log_handler_params = {"maxBytes":"100000"}
  e.g. 2
  log_handler_class = logging.handlers.TimedRotatingFileHandler
log_handler_params = {"backupCount":7,"interval":1,"when":"d"}
  default:
  log_handler_class = logging.FileHandler
log_handler_params = None

Tests

• ./run_unit_tests.sh python2.7.x and python3.5.x

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[mention-bot]

@ronfung, thanks for your PR! By analyzing the history of the files in this pull request, we identified @mistercrunch, @bolkedebruin and @aoen to be potential reviewers.

[criccomini]

@bolkedebruin @aoen thoughts on this?

[aoen]

I think we should close this, we are putting together a more detailed proposal for logging here:
#2422 which will include this functionality

[codecov-io]

Codecov Report

Merging #2452 into master will decrease coverage by 0.06%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #2452      +/-   ##
==========================================
- Coverage    69.4%   69.34%   -0.07%     
==========================================
  Files         146      146              
  Lines       11289    11299      +10     
==========================================
  Hits         7835     7835              
- Misses       3454     3464      +10

Impacted Files	Coverage Δ
airflow/bin/cli.py	51.07% <0%> (-0.8%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 751e936...86a4103. Read the comment docs.

[bolkedebruin]

Hi Chris, I agree with @aoen. @allisonwang is working in getting logging properly fixed which will allow you to this within the config more closely to the Python standards. Maybe you would like to chime in and help her/us getting it proper so we can work from there?

[bolkedebruin]

@ronfung can you close this please?