blob: b99417e9d6300dbd56cf6e8b72f4e9253ac1e0ab [file] [log] [blame]
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001#!/usr/bin/env python2
Tom Rini83d290c2018-05-06 17:58:06 -04002# SPDX-License-Identifier: GPL-2.0+
Masahiro Yamada5a27c732015-05-20 11:36:07 +09003#
4# Author: Masahiro Yamada <yamada.masahiro@socionext.com>
5#
Masahiro Yamada5a27c732015-05-20 11:36:07 +09006
7"""
8Move config options from headers to defconfig files.
9
10Since Kconfig was introduced to U-Boot, we have worked on moving
11config options from headers to Kconfig (defconfig).
12
13This tool intends to help this tremendous work.
14
15
16Usage
17-----
18
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090019First, you must edit the Kconfig to add the menu entries for the configs
Joe Hershberger96464ba2015-05-19 13:21:17 -050020you are moving.
21
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090022And then run this tool giving CONFIG names you want to move.
23For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE,
24simply type as follows:
Masahiro Yamada5a27c732015-05-20 11:36:07 +090025
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090026 $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE
Masahiro Yamada5a27c732015-05-20 11:36:07 +090027
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090028The tool walks through all the defconfig files and move the given CONFIGs.
Masahiro Yamada5a27c732015-05-20 11:36:07 +090029
30The log is also displayed on the terminal.
31
Masahiro Yamada1d085562016-05-19 15:52:02 +090032The log is printed for each defconfig as follows:
Masahiro Yamada5a27c732015-05-20 11:36:07 +090033
Masahiro Yamada1d085562016-05-19 15:52:02 +090034<defconfig_name>
35 <action1>
36 <action2>
37 <action3>
38 ...
Masahiro Yamada5a27c732015-05-20 11:36:07 +090039
Masahiro Yamada1d085562016-05-19 15:52:02 +090040<defconfig_name> is the name of the defconfig.
41
42<action*> shows what the tool did for that defconfig.
Masahiro Yamadac21fc7e2016-08-21 16:12:36 +090043It looks like one of the following:
Masahiro Yamada5a27c732015-05-20 11:36:07 +090044
45 - Move 'CONFIG_... '
46 This config option was moved to the defconfig
47
Masahiro Yamadacc008292016-05-19 15:51:56 +090048 - CONFIG_... is not defined in Kconfig. Do nothing.
Masahiro Yamada916224c2016-08-22 22:18:21 +090049 The entry for this CONFIG was not found in Kconfig. The option is not
50 defined in the config header, either. So, this case can be just skipped.
51
52 - CONFIG_... is not defined in Kconfig (suspicious). Do nothing.
53 This option is defined in the config header, but its entry was not found
54 in Kconfig.
Masahiro Yamadacc008292016-05-19 15:51:56 +090055 There are two common cases:
56 - You forgot to create an entry for the CONFIG before running
57 this tool, or made a typo in a CONFIG passed to this tool.
58 - The entry was hidden due to unmet 'depends on'.
Masahiro Yamada916224c2016-08-22 22:18:21 +090059 The tool does not know if the result is reasonable, so please check it
60 manually.
Masahiro Yamada5a27c732015-05-20 11:36:07 +090061
Masahiro Yamadacc008292016-05-19 15:51:56 +090062 - 'CONFIG_...' is the same as the define in Kconfig. Do nothing.
63 The define in the config header matched the one in Kconfig.
64 We do not need to touch it.
Masahiro Yamada5a27c732015-05-20 11:36:07 +090065
Masahiro Yamada90ed6cb2016-05-19 15:51:53 +090066 - Compiler is missing. Do nothing.
67 The compiler specified for this architecture was not found
68 in your PATH environment.
69 (If -e option is passed, the tool exits immediately.)
70
71 - Failed to process.
Masahiro Yamada5a27c732015-05-20 11:36:07 +090072 An error occurred during processing this defconfig. Skipped.
73 (If -e option is passed, the tool exits immediately on error.)
74
75Finally, you will be asked, Clean up headers? [y/n]:
76
77If you say 'y' here, the unnecessary config defines are removed
78from the config headers (include/configs/*.h).
79It just uses the regex method, so you should not rely on it.
80Just in case, please do 'git diff' to see what happened.
81
82
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090083How does it work?
84-----------------
Masahiro Yamada5a27c732015-05-20 11:36:07 +090085
86This tool runs configuration and builds include/autoconf.mk for every
87defconfig. The config options defined in Kconfig appear in the .config
88file (unless they are hidden because of unmet dependency.)
89On the other hand, the config options defined by board headers are seen
90in include/autoconf.mk. The tool looks for the specified options in both
Masahiro Yamadab6ef3932016-05-19 15:51:58 +090091of them to decide the appropriate action for the options. If the given
92config option is found in the .config, but its value does not match the
93one from the board header, the config option in the .config is replaced
94with the define in the board header. Then, the .config is synced by
95"make savedefconfig" and the defconfig is updated with it.
Masahiro Yamada5a27c732015-05-20 11:36:07 +090096
97For faster processing, this tool handles multi-threading. It creates
98separate build directories where the out-of-tree build is run. The
99temporary build directories are automatically created and deleted as
100needed. The number of threads are chosen based on the number of the CPU
101cores of your system although you can change it via -j (--jobs) option.
102
103
104Toolchains
105----------
106
107Appropriate toolchain are necessary to generate include/autoconf.mk
108for all the architectures supported by U-Boot. Most of them are available
Simon Glass6821a742017-07-10 14:47:47 -0600109at the kernel.org site, some are not provided by kernel.org. This tool uses
110the same tools as buildman, so see that tool for setup (e.g. --fetch-arch).
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900111
112
Simon Glassddf91c62017-06-01 19:39:00 -0600113Tips and trips
114--------------
115
116To sync only X86 defconfigs:
117
118 ./tools/moveconfig.py -s -d <(grep -l X86 configs/*)
119
120or:
121
122 grep -l X86 configs/* | ./tools/moveconfig.py -s -d -
123
124To process CONFIG_CMD_FPGAD only for a subset of configs based on path match:
125
126 ls configs/{hrcon*,iocon*,strider*} | \
127 ./tools/moveconfig.py -Cy CONFIG_CMD_FPGAD -d -
128
129
Simon Glass99b66602017-06-01 19:39:03 -0600130Finding implied CONFIGs
131-----------------------
132
133Some CONFIG options can be implied by others and this can help to reduce
134the size of the defconfig files. For example, CONFIG_X86 implies
135CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and
136all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to
137each of the x86 defconfig files.
138
139This tool can help find such configs. To use it, first build a database:
140
141 ./tools/moveconfig.py -b
142
143Then try to query it:
144
145 ./tools/moveconfig.py -i CONFIG_CMD_IRQ
146 CONFIG_CMD_IRQ found in 311/2384 defconfigs
147 44 : CONFIG_SYS_FSL_ERRATUM_IFC_A002769
148 41 : CONFIG_SYS_FSL_ERRATUM_A007075
149 31 : CONFIG_SYS_FSL_DDR_VER_44
150 28 : CONFIG_ARCH_P1010
151 28 : CONFIG_SYS_FSL_ERRATUM_P1010_A003549
152 28 : CONFIG_SYS_FSL_ERRATUM_SEC_A003571
153 28 : CONFIG_SYS_FSL_ERRATUM_IFC_A003399
154 25 : CONFIG_SYS_FSL_ERRATUM_A008044
155 22 : CONFIG_ARCH_P1020
156 21 : CONFIG_SYS_FSL_DDR_VER_46
157 20 : CONFIG_MAX_PIRQ_LINKS
158 20 : CONFIG_HPET_ADDRESS
159 20 : CONFIG_X86
160 20 : CONFIG_PCIE_ECAM_SIZE
161 20 : CONFIG_IRQ_SLOT_COUNT
162 20 : CONFIG_I8259_PIC
163 20 : CONFIG_CPU_ADDR_BITS
164 20 : CONFIG_RAMBASE
165 20 : CONFIG_SYS_FSL_ERRATUM_A005871
166 20 : CONFIG_PCIE_ECAM_BASE
167 20 : CONFIG_X86_TSC_TIMER
168 20 : CONFIG_I8254_TIMER
169 20 : CONFIG_CMD_GETTIME
170 19 : CONFIG_SYS_FSL_ERRATUM_A005812
171 18 : CONFIG_X86_RUN_32BIT
172 17 : CONFIG_CMD_CHIP_CONFIG
173 ...
174
175This shows a list of config options which might imply CONFIG_CMD_EEPROM along
176with how many defconfigs they cover. From this you can see that CONFIG_X86
177implies CONFIG_CMD_EEPROM. Therefore, instead of adding CONFIG_CMD_EEPROM to
178the defconfig of every x86 board, you could add a single imply line to the
179Kconfig file:
180
181 config X86
182 bool "x86 architecture"
183 ...
184 imply CMD_EEPROM
185
186That will cover 20 defconfigs. Many of the options listed are not suitable as
187they are not related. E.g. it would be odd for CONFIG_CMD_GETTIME to imply
188CMD_EEPROM.
189
190Using this search you can reduce the size of moveconfig patches.
191
Simon Glasscb008832017-06-15 21:39:33 -0600192You can automatically add 'imply' statements in the Kconfig with the -a
193option:
194
195 ./tools/moveconfig.py -s -i CONFIG_SCSI \
196 -a CONFIG_ARCH_LS1021A,CONFIG_ARCH_LS1043A
197
198This will add 'imply SCSI' to the two CONFIG options mentioned, assuming that
199the database indicates that they do actually imply CONFIG_SCSI and do not
200already have an 'imply SCSI'.
201
202The output shows where the imply is added:
203
204 18 : CONFIG_ARCH_LS1021A arch/arm/cpu/armv7/ls102xa/Kconfig:1
205 13 : CONFIG_ARCH_LS1043A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:11
206 12 : CONFIG_ARCH_LS1046A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:31
207
208The first number is the number of boards which can avoid having a special
209CONFIG_SCSI option in their defconfig file if this 'imply' is added.
210The location at the right is the Kconfig file and line number where the config
211appears. For example, adding 'imply CONFIG_SCSI' to the 'config ARCH_LS1021A'
212in arch/arm/cpu/armv7/ls102xa/Kconfig at line 1 will help 18 boards to reduce
213the size of their defconfig files.
214
215If you want to add an 'imply' to every imply config in the list, you can use
216
217 ./tools/moveconfig.py -s -i CONFIG_SCSI -a all
218
219To control which ones are displayed, use -I <list> where list is a list of
220options (use '-I help' to see possible options and their meaning).
221
222To skip showing you options that already have an 'imply' attached, use -A.
223
224When you have finished adding 'imply' options you can regenerate the
225defconfig files for affected boards with something like:
226
227 git show --stat | ./tools/moveconfig.py -s -d -
228
229This will regenerate only those defconfigs changed in the current commit.
230If you start with (say) 100 defconfigs being changed in the commit, and add
231a few 'imply' options as above, then regenerate, hopefully you can reduce the
232number of defconfigs changed in the commit.
233
Simon Glass99b66602017-06-01 19:39:03 -0600234
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900235Available options
236-----------------
237
238 -c, --color
239 Surround each portion of the log with escape sequences to display it
240 in color on the terminal.
241
Simon Glass9ede2122016-09-12 23:18:21 -0600242 -C, --commit
243 Create a git commit with the changes when the operation is complete. A
244 standard commit message is used which may need to be edited.
245
Joe Hershberger91040e82015-05-19 13:21:19 -0500246 -d, --defconfigs
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900247 Specify a file containing a list of defconfigs to move. The defconfig
Simon Glassddf91c62017-06-01 19:39:00 -0600248 files can be given with shell-style wildcards. Use '-' to read from stdin.
Joe Hershberger91040e82015-05-19 13:21:19 -0500249
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900250 -n, --dry-run
Masahiro Yamadab6ef3932016-05-19 15:51:58 +0900251 Perform a trial run that does not make any changes. It is useful to
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900252 see what is going to happen before one actually runs it.
253
254 -e, --exit-on-error
255 Exit immediately if Make exits with a non-zero status while processing
256 a defconfig file.
257
Masahiro Yamada8513dc02016-05-19 15:52:08 +0900258 -s, --force-sync
259 Do "make savedefconfig" forcibly for all the defconfig files.
260 If not specified, "make savedefconfig" only occurs for cases
261 where at least one CONFIG was moved.
262
Masahiro Yamada07913d12016-08-22 22:18:22 +0900263 -S, --spl
264 Look for moved config options in spl/include/autoconf.mk instead of
265 include/autoconf.mk. This is useful for moving options for SPL build
266 because SPL related options (mostly prefixed with CONFIG_SPL_) are
267 sometimes blocked by CONFIG_SPL_BUILD ifdef conditionals.
268
Joe Hershberger2144f882015-05-19 13:21:20 -0500269 -H, --headers-only
270 Only cleanup the headers; skip the defconfig processing
271
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900272 -j, --jobs
273 Specify the number of threads to run simultaneously. If not specified,
274 the number of threads is the same as the number of CPU cores.
275
Joe Hershberger6b96c1a2016-06-10 14:53:32 -0500276 -r, --git-ref
277 Specify the git ref to clone for building the autoconf.mk. If unspecified
278 use the CWD. This is useful for when changes to the Kconfig affect the
279 default values and you want to capture the state of the defconfig from
280 before that change was in effect. If in doubt, specify a ref pre-Kconfig
281 changes (use HEAD if Kconfig changes are not committed). Worst case it will
282 take a bit longer to run, but will always do the right thing.
283
Joe Hershberger95bf9c72015-05-19 13:21:24 -0500284 -v, --verbose
285 Show any build errors as boards are built
286
Simon Glass6b403df2016-09-12 23:18:20 -0600287 -y, --yes
288 Instead of prompting, automatically go ahead with all operations. This
Simon Glassddf91c62017-06-01 19:39:00 -0600289 includes cleaning up headers, CONFIG_SYS_EXTRA_OPTIONS, the config whitelist
290 and the README.
Simon Glass6b403df2016-09-12 23:18:20 -0600291
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900292To see the complete list of supported options, run
293
294 $ tools/moveconfig.py -h
295
296"""
297
Simon Glass99b66602017-06-01 19:39:03 -0600298import collections
Masahiro Yamada8ba1f5d2016-07-25 19:15:24 +0900299import copy
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900300import difflib
Masahiro Yamadac8e1b102016-05-19 15:52:07 +0900301import filecmp
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900302import fnmatch
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900303import glob
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900304import multiprocessing
305import optparse
306import os
Simon Glassd73fcb12017-06-01 19:39:02 -0600307import Queue
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900308import re
309import shutil
310import subprocess
311import sys
312import tempfile
Simon Glassd73fcb12017-06-01 19:39:02 -0600313import threading
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900314import time
315
Simon Glasscb008832017-06-15 21:39:33 -0600316sys.path.append(os.path.join(os.path.dirname(__file__), 'buildman'))
Simon Glass6821a742017-07-10 14:47:47 -0600317sys.path.append(os.path.join(os.path.dirname(__file__), 'patman'))
318import bsettings
Simon Glasscb008832017-06-15 21:39:33 -0600319import kconfiglib
Simon Glass6821a742017-07-10 14:47:47 -0600320import toolchain
Simon Glasscb008832017-06-15 21:39:33 -0600321
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900322SHOW_GNU_MAKE = 'scripts/show-gnu-make'
323SLEEP_TIME=0.03
324
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900325STATE_IDLE = 0
326STATE_DEFCONFIG = 1
327STATE_AUTOCONF = 2
Joe Hershberger96464ba2015-05-19 13:21:17 -0500328STATE_SAVEDEFCONFIG = 3
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900329
330ACTION_MOVE = 0
Masahiro Yamadacc008292016-05-19 15:51:56 +0900331ACTION_NO_ENTRY = 1
Masahiro Yamada916224c2016-08-22 22:18:21 +0900332ACTION_NO_ENTRY_WARN = 2
333ACTION_NO_CHANGE = 3
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900334
335COLOR_BLACK = '0;30'
336COLOR_RED = '0;31'
337COLOR_GREEN = '0;32'
338COLOR_BROWN = '0;33'
339COLOR_BLUE = '0;34'
340COLOR_PURPLE = '0;35'
341COLOR_CYAN = '0;36'
342COLOR_LIGHT_GRAY = '0;37'
343COLOR_DARK_GRAY = '1;30'
344COLOR_LIGHT_RED = '1;31'
345COLOR_LIGHT_GREEN = '1;32'
346COLOR_YELLOW = '1;33'
347COLOR_LIGHT_BLUE = '1;34'
348COLOR_LIGHT_PURPLE = '1;35'
349COLOR_LIGHT_CYAN = '1;36'
350COLOR_WHITE = '1;37'
351
Simon Glassf3b8e642017-06-01 19:39:01 -0600352AUTO_CONF_PATH = 'include/config/auto.conf'
Simon Glassd73fcb12017-06-01 19:39:02 -0600353CONFIG_DATABASE = 'moveconfig.db'
Simon Glassf3b8e642017-06-01 19:39:01 -0600354
Simon Glasscb008832017-06-15 21:39:33 -0600355CONFIG_LEN = len('CONFIG_')
Simon Glassf3b8e642017-06-01 19:39:01 -0600356
Markus Klotzbuecherb237d352019-05-15 15:15:52 +0200357SIZES = {
358 "SZ_1": 0x00000001, "SZ_2": 0x00000002,
359 "SZ_4": 0x00000004, "SZ_8": 0x00000008,
360 "SZ_16": 0x00000010, "SZ_32": 0x00000020,
361 "SZ_64": 0x00000040, "SZ_128": 0x00000080,
362 "SZ_256": 0x00000100, "SZ_512": 0x00000200,
363 "SZ_1K": 0x00000400, "SZ_2K": 0x00000800,
364 "SZ_4K": 0x00001000, "SZ_8K": 0x00002000,
365 "SZ_16K": 0x00004000, "SZ_32K": 0x00008000,
366 "SZ_64K": 0x00010000, "SZ_128K": 0x00020000,
367 "SZ_256K": 0x00040000, "SZ_512K": 0x00080000,
368 "SZ_1M": 0x00100000, "SZ_2M": 0x00200000,
369 "SZ_4M": 0x00400000, "SZ_8M": 0x00800000,
370 "SZ_16M": 0x01000000, "SZ_32M": 0x02000000,
371 "SZ_64M": 0x04000000, "SZ_128M": 0x08000000,
372 "SZ_256M": 0x10000000, "SZ_512M": 0x20000000,
373 "SZ_1G": 0x40000000, "SZ_2G": 0x80000000,
374 "SZ_4G": 0x100000000
375}
376
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900377### helper functions ###
378def get_devnull():
379 """Get the file object of '/dev/null' device."""
380 try:
381 devnull = subprocess.DEVNULL # py3k
382 except AttributeError:
383 devnull = open(os.devnull, 'wb')
384 return devnull
385
386def check_top_directory():
387 """Exit if we are not at the top of source directory."""
388 for f in ('README', 'Licenses'):
389 if not os.path.exists(f):
390 sys.exit('Please run at the top of source directory.')
391
Masahiro Yamadabd63e5b2016-05-19 15:51:54 +0900392def check_clean_directory():
393 """Exit if the source tree is not clean."""
394 for f in ('.config', 'include/config'):
395 if os.path.exists(f):
396 sys.exit("source tree is not clean, please run 'make mrproper'")
397
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900398def get_make_cmd():
399 """Get the command name of GNU Make.
400
401 U-Boot needs GNU Make for building, but the command name is not
402 necessarily "make". (for example, "gmake" on FreeBSD).
403 Returns the most appropriate command name on your system.
404 """
405 process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE)
406 ret = process.communicate()
407 if process.returncode:
408 sys.exit('GNU Make not found')
409 return ret[0].rstrip()
410
Simon Glass25f978c2017-06-01 19:38:58 -0600411def get_matched_defconfig(line):
412 """Get the defconfig files that match a pattern
413
414 Args:
415 line: Path or filename to match, e.g. 'configs/snow_defconfig' or
416 'k2*_defconfig'. If no directory is provided, 'configs/' is
417 prepended
418
419 Returns:
420 a list of matching defconfig files
421 """
422 dirname = os.path.dirname(line)
423 if dirname:
424 pattern = line
425 else:
426 pattern = os.path.join('configs', line)
427 return glob.glob(pattern) + glob.glob(pattern + '_defconfig')
428
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900429def get_matched_defconfigs(defconfigs_file):
Simon Glassee4e61b2017-06-01 19:38:59 -0600430 """Get all the defconfig files that match the patterns in a file.
431
432 Args:
433 defconfigs_file: File containing a list of defconfigs to process, or
434 '-' to read the list from stdin
435
436 Returns:
437 A list of paths to defconfig files, with no duplicates
438 """
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900439 defconfigs = []
Simon Glassee4e61b2017-06-01 19:38:59 -0600440 if defconfigs_file == '-':
441 fd = sys.stdin
442 defconfigs_file = 'stdin'
443 else:
444 fd = open(defconfigs_file)
445 for i, line in enumerate(fd):
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900446 line = line.strip()
447 if not line:
448 continue # skip blank lines silently
Simon Glass2ddd85d2017-06-15 21:39:31 -0600449 if ' ' in line:
450 line = line.split(' ')[0] # handle 'git log' input
Simon Glass25f978c2017-06-01 19:38:58 -0600451 matched = get_matched_defconfig(line)
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +0900452 if not matched:
453 print >> sys.stderr, "warning: %s:%d: no defconfig matched '%s'" % \
454 (defconfigs_file, i + 1, line)
455
456 defconfigs += matched
457
458 # use set() to drop multiple matching
459 return [ defconfig[len('configs') + 1:] for defconfig in set(defconfigs) ]
460
Masahiro Yamada684c3062016-07-25 19:15:28 +0900461def get_all_defconfigs():
462 """Get all the defconfig files under the configs/ directory."""
463 defconfigs = []
464 for (dirpath, dirnames, filenames) in os.walk('configs'):
465 dirpath = dirpath[len('configs') + 1:]
466 for filename in fnmatch.filter(filenames, '*_defconfig'):
467 defconfigs.append(os.path.join(dirpath, filename))
468
469 return defconfigs
470
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900471def color_text(color_enabled, color, string):
472 """Return colored string."""
473 if color_enabled:
Masahiro Yamada1d085562016-05-19 15:52:02 +0900474 # LF should not be surrounded by the escape sequence.
475 # Otherwise, additional whitespace or line-feed might be printed.
476 return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else ''
477 for s in string.split('\n') ])
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900478 else:
479 return string
480
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900481def show_diff(a, b, file_path, color_enabled):
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900482 """Show unidified diff.
483
484 Arguments:
485 a: A list of lines (before)
486 b: A list of lines (after)
487 file_path: Path to the file
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900488 color_enabled: Display the diff in color
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900489 """
490
491 diff = difflib.unified_diff(a, b,
492 fromfile=os.path.join('a', file_path),
493 tofile=os.path.join('b', file_path))
494
495 for line in diff:
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900496 if line[0] == '-' and line[1] != '-':
497 print color_text(color_enabled, COLOR_RED, line),
498 elif line[0] == '+' and line[1] != '+':
499 print color_text(color_enabled, COLOR_GREEN, line),
500 else:
501 print line,
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900502
Masahiro Yamada8ba1f5d2016-07-25 19:15:24 +0900503def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre,
504 extend_post):
505 """Extend matched lines if desired patterns are found before/after already
506 matched lines.
507
508 Arguments:
509 lines: A list of lines handled.
510 matched: A list of line numbers that have been already matched.
511 (will be updated by this function)
512 pre_patterns: A list of regular expression that should be matched as
513 preamble.
514 post_patterns: A list of regular expression that should be matched as
515 postamble.
516 extend_pre: Add the line number of matched preamble to the matched list.
517 extend_post: Add the line number of matched postamble to the matched list.
518 """
519 extended_matched = []
520
521 j = matched[0]
522
523 for i in matched:
524 if i == 0 or i < j:
525 continue
526 j = i
527 while j in matched:
528 j += 1
529 if j >= len(lines):
530 break
531
532 for p in pre_patterns:
533 if p.search(lines[i - 1]):
534 break
535 else:
536 # not matched
537 continue
538
539 for p in post_patterns:
540 if p.search(lines[j]):
541 break
542 else:
543 # not matched
544 continue
545
546 if extend_pre:
547 extended_matched.append(i - 1)
548 if extend_post:
549 extended_matched.append(j)
550
551 matched += extended_matched
552 matched.sort()
553
Chris Packham85edfc12017-05-02 21:30:46 +1200554def confirm(options, prompt):
555 if not options.yes:
556 while True:
557 choice = raw_input('{} [y/n]: '.format(prompt))
558 choice = choice.lower()
559 print choice
560 if choice == 'y' or choice == 'n':
561 break
562
563 if choice == 'n':
564 return False
565
566 return True
567
Chris Packham4d9dbb12019-01-30 20:23:16 +1300568def cleanup_empty_blocks(header_path, options):
569 """Clean up empty conditional blocks
570
571 Arguments:
572 header_path: path to the cleaned file.
573 options: option flags.
574 """
575 pattern = re.compile(r'^\s*#\s*if.*$\n^\s*#\s*endif.*$\n*', flags=re.M)
576 with open(header_path) as f:
577 data = f.read()
578
579 new_data = pattern.sub('\n', data)
580
581 show_diff(data.splitlines(True), new_data.splitlines(True), header_path,
582 options.color)
583
584 if options.dry_run:
585 return
586
587 with open(header_path, 'w') as f:
588 f.write(new_data)
589
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900590def cleanup_one_header(header_path, patterns, options):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900591 """Clean regex-matched lines away from a file.
592
593 Arguments:
594 header_path: path to the cleaned file.
595 patterns: list of regex patterns. Any lines matching to these
596 patterns are deleted.
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900597 options: option flags.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900598 """
599 with open(header_path) as f:
600 lines = f.readlines()
601
602 matched = []
603 for i, line in enumerate(lines):
Masahiro Yamadaa3a779f2016-07-25 19:15:27 +0900604 if i - 1 in matched and lines[i - 1][-2:] == '\\\n':
605 matched.append(i)
606 continue
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900607 for pattern in patterns:
Masahiro Yamada8ba1f5d2016-07-25 19:15:24 +0900608 if pattern.search(line):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900609 matched.append(i)
610 break
611
Masahiro Yamada8ba1f5d2016-07-25 19:15:24 +0900612 if not matched:
613 return
614
615 # remove empty #ifdef ... #endif, successive blank lines
616 pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef
617 pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else
618 pattern_endif = re.compile(r'#\s*endif\W') # #endif
619 pattern_blank = re.compile(r'^\s*$') # empty line
620
621 while True:
622 old_matched = copy.copy(matched)
623 extend_matched_lines(lines, matched, [pattern_if],
624 [pattern_endif], True, True)
625 extend_matched_lines(lines, matched, [pattern_elif],
626 [pattern_elif, pattern_endif], True, False)
627 extend_matched_lines(lines, matched, [pattern_if, pattern_elif],
628 [pattern_blank], False, True)
629 extend_matched_lines(lines, matched, [pattern_blank],
630 [pattern_elif, pattern_endif], True, False)
631 extend_matched_lines(lines, matched, [pattern_blank],
632 [pattern_blank], True, False)
633 if matched == old_matched:
634 break
635
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900636 tolines = copy.copy(lines)
637
638 for i in reversed(matched):
639 tolines.pop(i)
640
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900641 show_diff(lines, tolines, header_path, options.color)
Masahiro Yamada8ba1f5d2016-07-25 19:15:24 +0900642
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900643 if options.dry_run:
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900644 return
645
646 with open(header_path, 'w') as f:
Masahiro Yamadaf2f69812016-07-25 19:15:25 +0900647 for line in tolines:
648 f.write(line)
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900649
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900650def cleanup_headers(configs, options):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900651 """Delete config defines from board headers.
652
653 Arguments:
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900654 configs: A list of CONFIGs to remove.
Masahiro Yamadae9ea1222016-07-25 19:15:26 +0900655 options: option flags.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900656 """
Chris Packham85edfc12017-05-02 21:30:46 +1200657 if not confirm(options, 'Clean up headers?'):
658 return
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900659
660 patterns = []
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900661 for config in configs:
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900662 patterns.append(re.compile(r'#\s*define\s+%s\W' % config))
663 patterns.append(re.compile(r'#\s*undef\s+%s\W' % config))
664
Joe Hershberger60727f52015-05-19 13:21:21 -0500665 for dir in 'include', 'arch', 'board':
666 for (dirpath, dirnames, filenames) in os.walk(dir):
Masahiro Yamadadc6de502016-07-25 19:15:22 +0900667 if dirpath == os.path.join('include', 'generated'):
668 continue
Joe Hershberger60727f52015-05-19 13:21:21 -0500669 for filename in filenames:
670 if not fnmatch.fnmatch(filename, '*~'):
Chris Packham4d9dbb12019-01-30 20:23:16 +1300671 header_path = os.path.join(dirpath, filename)
672 cleanup_one_header(header_path, patterns, options)
673 cleanup_empty_blocks(header_path, options)
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900674
Masahiro Yamada9ab02962016-07-25 19:15:29 +0900675def cleanup_one_extra_option(defconfig_path, configs, options):
676 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file.
677
678 Arguments:
679 defconfig_path: path to the cleaned defconfig file.
680 configs: A list of CONFIGs to remove.
681 options: option flags.
682 """
683
684 start = 'CONFIG_SYS_EXTRA_OPTIONS="'
685 end = '"\n'
686
687 with open(defconfig_path) as f:
688 lines = f.readlines()
689
690 for i, line in enumerate(lines):
691 if line.startswith(start) and line.endswith(end):
692 break
693 else:
694 # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig
695 return
696
697 old_tokens = line[len(start):-len(end)].split(',')
698 new_tokens = []
699
700 for token in old_tokens:
701 pos = token.find('=')
702 if not (token[:pos] if pos >= 0 else token) in configs:
703 new_tokens.append(token)
704
705 if new_tokens == old_tokens:
706 return
707
708 tolines = copy.copy(lines)
709
710 if new_tokens:
711 tolines[i] = start + ','.join(new_tokens) + end
712 else:
713 tolines.pop(i)
714
715 show_diff(lines, tolines, defconfig_path, options.color)
716
717 if options.dry_run:
718 return
719
720 with open(defconfig_path, 'w') as f:
721 for line in tolines:
722 f.write(line)
723
724def cleanup_extra_options(configs, options):
725 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files.
726
727 Arguments:
728 configs: A list of CONFIGs to remove.
729 options: option flags.
730 """
Chris Packham85edfc12017-05-02 21:30:46 +1200731 if not confirm(options, 'Clean up CONFIG_SYS_EXTRA_OPTIONS?'):
732 return
Masahiro Yamada9ab02962016-07-25 19:15:29 +0900733
734 configs = [ config[len('CONFIG_'):] for config in configs ]
735
736 defconfigs = get_all_defconfigs()
737
738 for defconfig in defconfigs:
739 cleanup_one_extra_option(os.path.join('configs', defconfig), configs,
740 options)
741
Chris Packhamca438342017-05-02 21:30:47 +1200742def cleanup_whitelist(configs, options):
743 """Delete config whitelist entries
744
745 Arguments:
746 configs: A list of CONFIGs to remove.
747 options: option flags.
748 """
749 if not confirm(options, 'Clean up whitelist entries?'):
750 return
751
752 with open(os.path.join('scripts', 'config_whitelist.txt')) as f:
753 lines = f.readlines()
754
755 lines = [x for x in lines if x.strip() not in configs]
756
757 with open(os.path.join('scripts', 'config_whitelist.txt'), 'w') as f:
758 f.write(''.join(lines))
759
Chris Packhamf90df592017-05-02 21:30:48 +1200760def find_matching(patterns, line):
761 for pat in patterns:
762 if pat.search(line):
763 return True
764 return False
765
766def cleanup_readme(configs, options):
767 """Delete config description in README
768
769 Arguments:
770 configs: A list of CONFIGs to remove.
771 options: option flags.
772 """
773 if not confirm(options, 'Clean up README?'):
774 return
775
776 patterns = []
777 for config in configs:
778 patterns.append(re.compile(r'^\s+%s' % config))
779
780 with open('README') as f:
781 lines = f.readlines()
782
783 found = False
784 newlines = []
785 for line in lines:
786 if not found:
787 found = find_matching(patterns, line)
788 if found:
789 continue
790
791 if found and re.search(r'^\s+CONFIG', line):
792 found = False
793
794 if not found:
795 newlines.append(line)
796
797 with open('README', 'w') as f:
798 f.write(''.join(newlines))
799
Markus Klotzbuecherb237d352019-05-15 15:15:52 +0200800def try_expand(line):
801 """If value looks like an expression, try expanding it
802 Otherwise just return the existing value
803 """
804 if line.find('=') == -1:
805 return line
806
807 try:
808 cfg, val = re.split("=", line)
809 val= val.strip('\"')
810 if re.search("[*+-/]|<<|SZ_+|\(([^\)]+)\)", val):
811 newval = hex(eval(val, SIZES))
812 print "\tExpanded expression %s to %s" % (val, newval)
813 return cfg+'='+newval
814 except:
815 print "\tFailed to expand expression in %s" % line
816
817 return line
818
Chris Packhamca438342017-05-02 21:30:47 +1200819
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900820### classes ###
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +0900821class Progress:
822
823 """Progress Indicator"""
824
825 def __init__(self, total):
826 """Create a new progress indicator.
827
828 Arguments:
829 total: A number of defconfig files to process.
830 """
831 self.current = 0
832 self.total = total
833
834 def inc(self):
835 """Increment the number of processed defconfig files."""
836
837 self.current += 1
838
839 def show(self):
840 """Display the progress."""
841 print ' %d defconfigs out of %d\r' % (self.current, self.total),
842 sys.stdout.flush()
843
Simon Glasscb008832017-06-15 21:39:33 -0600844
845class KconfigScanner:
846 """Kconfig scanner."""
847
848 def __init__(self):
849 """Scan all the Kconfig files and create a Config object."""
850 # Define environment variables referenced from Kconfig
851 os.environ['srctree'] = os.getcwd()
852 os.environ['UBOOTVERSION'] = 'dummy'
853 os.environ['KCONFIG_OBJDIR'] = ''
Tom Rini65e05dd2019-09-20 17:42:09 -0400854 self.conf = kconfiglib.Kconfig()
Simon Glasscb008832017-06-15 21:39:33 -0600855
856
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900857class KconfigParser:
858
859 """A parser of .config and include/autoconf.mk."""
860
861 re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"')
862 re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"')
863
Masahiro Yamada522e8dc2016-05-19 15:52:01 +0900864 def __init__(self, configs, options, build_dir):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900865 """Create a new parser.
866
867 Arguments:
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900868 configs: A list of CONFIGs to move.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900869 options: option flags.
870 build_dir: Build directory.
871 """
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900872 self.configs = configs
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900873 self.options = options
Masahiro Yamada1f169922016-05-19 15:52:00 +0900874 self.dotconfig = os.path.join(build_dir, '.config')
875 self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk')
Masahiro Yamada07913d12016-08-22 22:18:22 +0900876 self.spl_autoconf = os.path.join(build_dir, 'spl', 'include',
877 'autoconf.mk')
Simon Glassf3b8e642017-06-01 19:39:01 -0600878 self.config_autoconf = os.path.join(build_dir, AUTO_CONF_PATH)
Masahiro Yamada5da4f852016-05-19 15:52:06 +0900879 self.defconfig = os.path.join(build_dir, 'defconfig')
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900880
Simon Glass6821a742017-07-10 14:47:47 -0600881 def get_arch(self):
882 """Parse .config file and return the architecture.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900883
884 Returns:
Simon Glass6821a742017-07-10 14:47:47 -0600885 Architecture name (e.g. 'arm').
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900886 """
887 arch = ''
888 cpu = ''
Masahiro Yamada1f169922016-05-19 15:52:00 +0900889 for line in open(self.dotconfig):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900890 m = self.re_arch.match(line)
891 if m:
892 arch = m.group(1)
893 continue
894 m = self.re_cpu.match(line)
895 if m:
896 cpu = m.group(1)
897
Masahiro Yamada90ed6cb2016-05-19 15:51:53 +0900898 if not arch:
899 return None
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900900
901 # fix-up for aarch64
902 if arch == 'arm' and cpu == 'armv8':
903 arch = 'aarch64'
904
Simon Glass6821a742017-07-10 14:47:47 -0600905 return arch
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900906
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900907 def parse_one_config(self, config, dotconfig_lines, autoconf_lines):
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900908 """Parse .config, defconfig, include/autoconf.mk for one config.
909
910 This function looks for the config options in the lines from
911 defconfig, .config, and include/autoconf.mk in order to decide
912 which action should be taken for this defconfig.
913
914 Arguments:
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900915 config: CONFIG name to parse.
Masahiro Yamadacc008292016-05-19 15:51:56 +0900916 dotconfig_lines: lines from the .config file.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900917 autoconf_lines: lines from the include/autoconf.mk file.
918
919 Returns:
920 A tupple of the action for this defconfig and the line
921 matched for the config.
922 """
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900923 not_set = '# %s is not set' % config
924
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900925 for line in autoconf_lines:
926 line = line.rstrip()
927 if line.startswith(config + '='):
Masahiro Yamadacc008292016-05-19 15:51:56 +0900928 new_val = line
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900929 break
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900930 else:
Masahiro Yamadacc008292016-05-19 15:51:56 +0900931 new_val = not_set
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900932
Markus Klotzbuecherb237d352019-05-15 15:15:52 +0200933 new_val = try_expand(new_val)
934
Masahiro Yamada916224c2016-08-22 22:18:21 +0900935 for line in dotconfig_lines:
936 line = line.rstrip()
937 if line.startswith(config + '=') or line == not_set:
938 old_val = line
939 break
940 else:
941 if new_val == not_set:
942 return (ACTION_NO_ENTRY, config)
943 else:
944 return (ACTION_NO_ENTRY_WARN, config)
945
Masahiro Yamadacc008292016-05-19 15:51:56 +0900946 # If this CONFIG is neither bool nor trisate
947 if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set:
948 # tools/scripts/define2mk.sed changes '1' to 'y'.
949 # This is a problem if the CONFIG is int type.
950 # Check the type in Kconfig and handle it correctly.
951 if new_val[-2:] == '=y':
952 new_val = new_val[:-1] + '1'
953
Masahiro Yamada50301592016-06-15 14:33:50 +0900954 return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE,
955 new_val)
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900956
Masahiro Yamada1d085562016-05-19 15:52:02 +0900957 def update_dotconfig(self):
Masahiro Yamada6ff36d22016-05-19 15:51:50 +0900958 """Parse files for the config options and update the .config.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900959
Masahiro Yamadacc008292016-05-19 15:51:56 +0900960 This function parses the generated .config and include/autoconf.mk
961 searching the target options.
Masahiro Yamada6ff36d22016-05-19 15:51:50 +0900962 Move the config option(s) to the .config as needed.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900963
964 Arguments:
965 defconfig: defconfig name.
Masahiro Yamada522e8dc2016-05-19 15:52:01 +0900966
967 Returns:
Masahiro Yamada7fb0bac2016-05-19 15:52:04 +0900968 Return a tuple of (updated flag, log string).
969 The "updated flag" is True if the .config was updated, False
970 otherwise. The "log string" shows what happend to the .config.
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900971 """
972
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900973 results = []
Masahiro Yamada7fb0bac2016-05-19 15:52:04 +0900974 updated = False
Masahiro Yamada916224c2016-08-22 22:18:21 +0900975 suspicious = False
Masahiro Yamada07913d12016-08-22 22:18:22 +0900976 rm_files = [self.config_autoconf, self.autoconf]
977
978 if self.options.spl:
979 if os.path.exists(self.spl_autoconf):
980 autoconf_path = self.spl_autoconf
981 rm_files.append(self.spl_autoconf)
982 else:
983 for f in rm_files:
984 os.remove(f)
985 return (updated, suspicious,
986 color_text(self.options.color, COLOR_BROWN,
987 "SPL is not enabled. Skipped.") + '\n')
988 else:
989 autoconf_path = self.autoconf
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900990
Masahiro Yamada1f169922016-05-19 15:52:00 +0900991 with open(self.dotconfig) as f:
Masahiro Yamadacc008292016-05-19 15:51:56 +0900992 dotconfig_lines = f.readlines()
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900993
Masahiro Yamada07913d12016-08-22 22:18:22 +0900994 with open(autoconf_path) as f:
Masahiro Yamada5a27c732015-05-20 11:36:07 +0900995 autoconf_lines = f.readlines()
996
Masahiro Yamadab134bc12016-05-19 15:51:57 +0900997 for config in self.configs:
998 result = self.parse_one_config(config, dotconfig_lines,
Joe Hershberger96464ba2015-05-19 13:21:17 -0500999 autoconf_lines)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001000 results.append(result)
1001
1002 log = ''
1003
1004 for (action, value) in results:
1005 if action == ACTION_MOVE:
1006 actlog = "Move '%s'" % value
1007 log_color = COLOR_LIGHT_GREEN
Masahiro Yamadacc008292016-05-19 15:51:56 +09001008 elif action == ACTION_NO_ENTRY:
1009 actlog = "%s is not defined in Kconfig. Do nothing." % value
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001010 log_color = COLOR_LIGHT_BLUE
Masahiro Yamada916224c2016-08-22 22:18:21 +09001011 elif action == ACTION_NO_ENTRY_WARN:
1012 actlog = "%s is not defined in Kconfig (suspicious). Do nothing." % value
1013 log_color = COLOR_YELLOW
1014 suspicious = True
Masahiro Yamadacc008292016-05-19 15:51:56 +09001015 elif action == ACTION_NO_CHANGE:
1016 actlog = "'%s' is the same as the define in Kconfig. Do nothing." \
1017 % value
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001018 log_color = COLOR_LIGHT_PURPLE
Masahiro Yamada07913d12016-08-22 22:18:22 +09001019 elif action == ACTION_SPL_NOT_EXIST:
1020 actlog = "SPL is not enabled for this defconfig. Skip."
1021 log_color = COLOR_PURPLE
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001022 else:
1023 sys.exit("Internal Error. This should not happen.")
1024
Masahiro Yamada1d085562016-05-19 15:52:02 +09001025 log += color_text(self.options.color, log_color, actlog) + '\n'
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001026
Masahiro Yamada1f169922016-05-19 15:52:00 +09001027 with open(self.dotconfig, 'a') as f:
Masahiro Yamadae423d172016-05-19 15:51:49 +09001028 for (action, value) in results:
1029 if action == ACTION_MOVE:
1030 f.write(value + '\n')
Masahiro Yamada7fb0bac2016-05-19 15:52:04 +09001031 updated = True
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001032
Masahiro Yamada5da4f852016-05-19 15:52:06 +09001033 self.results = results
Masahiro Yamada07913d12016-08-22 22:18:22 +09001034 for f in rm_files:
1035 os.remove(f)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001036
Masahiro Yamada916224c2016-08-22 22:18:21 +09001037 return (updated, suspicious, log)
Masahiro Yamada522e8dc2016-05-19 15:52:01 +09001038
Masahiro Yamada5da4f852016-05-19 15:52:06 +09001039 def check_defconfig(self):
1040 """Check the defconfig after savedefconfig
1041
1042 Returns:
1043 Return additional log if moved CONFIGs were removed again by
1044 'make savedefconfig'.
1045 """
1046
1047 log = ''
1048
1049 with open(self.defconfig) as f:
1050 defconfig_lines = f.readlines()
1051
1052 for (action, value) in self.results:
1053 if action != ACTION_MOVE:
1054 continue
1055 if not value + '\n' in defconfig_lines:
1056 log += color_text(self.options.color, COLOR_YELLOW,
1057 "'%s' was removed by savedefconfig.\n" %
1058 value)
1059
1060 return log
1061
Simon Glassd73fcb12017-06-01 19:39:02 -06001062
1063class DatabaseThread(threading.Thread):
1064 """This thread processes results from Slot threads.
1065
1066 It collects the data in the master config directary. There is only one
1067 result thread, and this helps to serialise the build output.
1068 """
1069 def __init__(self, config_db, db_queue):
1070 """Set up a new result thread
1071
1072 Args:
1073 builder: Builder which will be sent each result
1074 """
1075 threading.Thread.__init__(self)
1076 self.config_db = config_db
1077 self.db_queue= db_queue
1078
1079 def run(self):
1080 """Called to start up the result thread.
1081
1082 We collect the next result job and pass it on to the build.
1083 """
1084 while True:
1085 defconfig, configs = self.db_queue.get()
1086 self.config_db[defconfig] = configs
1087 self.db_queue.task_done()
1088
1089
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001090class Slot:
1091
1092 """A slot to store a subprocess.
1093
1094 Each instance of this class handles one subprocess.
1095 This class is useful to control multiple threads
1096 for faster processing.
1097 """
1098
Simon Glass6821a742017-07-10 14:47:47 -06001099 def __init__(self, toolchains, configs, options, progress, devnull,
1100 make_cmd, reference_src_dir, db_queue):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001101 """Create a new process slot.
1102
1103 Arguments:
Simon Glass6821a742017-07-10 14:47:47 -06001104 toolchains: Toolchains object containing toolchains.
Masahiro Yamadab134bc12016-05-19 15:51:57 +09001105 configs: A list of CONFIGs to move.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001106 options: option flags.
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001107 progress: A progress indicator.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001108 devnull: A file object of '/dev/null'.
1109 make_cmd: command name of GNU Make.
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001110 reference_src_dir: Determine the true starting config state from this
1111 source tree.
Simon Glassd73fcb12017-06-01 19:39:02 -06001112 db_queue: output queue to write config info for the database
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001113 """
Simon Glass6821a742017-07-10 14:47:47 -06001114 self.toolchains = toolchains
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001115 self.options = options
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001116 self.progress = progress
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001117 self.build_dir = tempfile.mkdtemp()
1118 self.devnull = devnull
1119 self.make_cmd = (make_cmd, 'O=' + self.build_dir)
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001120 self.reference_src_dir = reference_src_dir
Simon Glassd73fcb12017-06-01 19:39:02 -06001121 self.db_queue = db_queue
Masahiro Yamada522e8dc2016-05-19 15:52:01 +09001122 self.parser = KconfigParser(configs, options, self.build_dir)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001123 self.state = STATE_IDLE
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001124 self.failed_boards = set()
1125 self.suspicious_boards = set()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001126
1127 def __del__(self):
1128 """Delete the working directory
1129
1130 This function makes sure the temporary directory is cleaned away
1131 even if Python suddenly dies due to error. It should be done in here
Joe Hershbergerf2dae752016-06-10 14:53:29 -05001132 because it is guaranteed the destructor is always invoked when the
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001133 instance of the class gets unreferenced.
1134
1135 If the subprocess is still running, wait until it finishes.
1136 """
1137 if self.state != STATE_IDLE:
1138 while self.ps.poll() == None:
1139 pass
1140 shutil.rmtree(self.build_dir)
1141
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001142 def add(self, defconfig):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001143 """Assign a new subprocess for defconfig and add it to the slot.
1144
1145 If the slot is vacant, create a new subprocess for processing the
1146 given defconfig and add it to the slot. Just returns False if
1147 the slot is occupied (i.e. the current subprocess is still running).
1148
1149 Arguments:
1150 defconfig: defconfig name.
1151
1152 Returns:
1153 Return True on success or False on failure
1154 """
1155 if self.state != STATE_IDLE:
1156 return False
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001157
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001158 self.defconfig = defconfig
Masahiro Yamada1d085562016-05-19 15:52:02 +09001159 self.log = ''
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001160 self.current_src_dir = self.reference_src_dir
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001161 self.do_defconfig()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001162 return True
1163
1164 def poll(self):
1165 """Check the status of the subprocess and handle it as needed.
1166
1167 Returns True if the slot is vacant (i.e. in idle state).
1168 If the configuration is successfully finished, assign a new
1169 subprocess to build include/autoconf.mk.
1170 If include/autoconf.mk is generated, invoke the parser to
Masahiro Yamada7fb0bac2016-05-19 15:52:04 +09001171 parse the .config and the include/autoconf.mk, moving
1172 config options to the .config as needed.
1173 If the .config was updated, run "make savedefconfig" to sync
1174 it, update the original defconfig, and then set the slot back
1175 to the idle state.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001176
1177 Returns:
1178 Return True if the subprocess is terminated, False otherwise
1179 """
1180 if self.state == STATE_IDLE:
1181 return True
1182
1183 if self.ps.poll() == None:
1184 return False
1185
1186 if self.ps.poll() != 0:
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001187 self.handle_error()
1188 elif self.state == STATE_DEFCONFIG:
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001189 if self.reference_src_dir and not self.current_src_dir:
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001190 self.do_savedefconfig()
1191 else:
1192 self.do_autoconf()
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001193 elif self.state == STATE_AUTOCONF:
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001194 if self.current_src_dir:
1195 self.current_src_dir = None
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001196 self.do_defconfig()
Simon Glassd73fcb12017-06-01 19:39:02 -06001197 elif self.options.build_db:
1198 self.do_build_db()
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001199 else:
1200 self.do_savedefconfig()
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001201 elif self.state == STATE_SAVEDEFCONFIG:
1202 self.update_defconfig()
1203 else:
1204 sys.exit("Internal Error. This should not happen.")
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001205
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001206 return True if self.state == STATE_IDLE else False
Joe Hershberger96464ba2015-05-19 13:21:17 -05001207
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001208 def handle_error(self):
1209 """Handle error cases."""
Masahiro Yamada8513dc02016-05-19 15:52:08 +09001210
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001211 self.log += color_text(self.options.color, COLOR_LIGHT_RED,
1212 "Failed to process.\n")
1213 if self.options.verbose:
1214 self.log += color_text(self.options.color, COLOR_LIGHT_CYAN,
1215 self.ps.stderr.read())
1216 self.finish(False)
Joe Hershberger96464ba2015-05-19 13:21:17 -05001217
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001218 def do_defconfig(self):
1219 """Run 'make <board>_defconfig' to create the .config file."""
Masahiro Yamadac8e1b102016-05-19 15:52:07 +09001220
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001221 cmd = list(self.make_cmd)
1222 cmd.append(self.defconfig)
1223 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001224 stderr=subprocess.PIPE,
1225 cwd=self.current_src_dir)
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001226 self.state = STATE_DEFCONFIG
Masahiro Yamadac8e1b102016-05-19 15:52:07 +09001227
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001228 def do_autoconf(self):
Simon Glassf3b8e642017-06-01 19:39:01 -06001229 """Run 'make AUTO_CONF_PATH'."""
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001230
Simon Glass6821a742017-07-10 14:47:47 -06001231 arch = self.parser.get_arch()
1232 try:
1233 toolchain = self.toolchains.Select(arch)
1234 except ValueError:
Masahiro Yamada1d085562016-05-19 15:52:02 +09001235 self.log += color_text(self.options.color, COLOR_YELLOW,
Chris Packhamce3ba452017-08-27 20:00:51 +12001236 "Tool chain for '%s' is missing. Do nothing.\n" % arch)
Masahiro Yamada4efef992016-05-19 15:52:03 +09001237 self.finish(False)
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001238 return
Simon Glass6821a742017-07-10 14:47:47 -06001239 env = toolchain.MakeEnvironment(False)
Masahiro Yamada90ed6cb2016-05-19 15:51:53 +09001240
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001241 cmd = list(self.make_cmd)
Joe Hershberger7740f652015-05-19 13:21:18 -05001242 cmd.append('KCONFIG_IGNORE_DUPLICATES=1')
Simon Glassf3b8e642017-06-01 19:39:01 -06001243 cmd.append(AUTO_CONF_PATH)
Simon Glass6821a742017-07-10 14:47:47 -06001244 self.ps = subprocess.Popen(cmd, stdout=self.devnull, env=env,
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001245 stderr=subprocess.PIPE,
1246 cwd=self.current_src_dir)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001247 self.state = STATE_AUTOCONF
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001248
Simon Glassd73fcb12017-06-01 19:39:02 -06001249 def do_build_db(self):
1250 """Add the board to the database"""
1251 configs = {}
1252 with open(os.path.join(self.build_dir, AUTO_CONF_PATH)) as fd:
1253 for line in fd.readlines():
1254 if line.startswith('CONFIG'):
1255 config, value = line.split('=', 1)
1256 configs[config] = value.rstrip()
1257 self.db_queue.put([self.defconfig, configs])
1258 self.finish(True)
1259
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001260 def do_savedefconfig(self):
1261 """Update the .config and run 'make savedefconfig'."""
1262
Masahiro Yamada916224c2016-08-22 22:18:21 +09001263 (updated, suspicious, log) = self.parser.update_dotconfig()
1264 if suspicious:
1265 self.suspicious_boards.add(self.defconfig)
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001266 self.log += log
1267
1268 if not self.options.force_sync and not updated:
1269 self.finish(True)
1270 return
1271 if updated:
1272 self.log += color_text(self.options.color, COLOR_LIGHT_GREEN,
1273 "Syncing by savedefconfig...\n")
1274 else:
1275 self.log += "Syncing by savedefconfig (forced by option)...\n"
1276
1277 cmd = list(self.make_cmd)
1278 cmd.append('savedefconfig')
1279 self.ps = subprocess.Popen(cmd, stdout=self.devnull,
1280 stderr=subprocess.PIPE)
1281 self.state = STATE_SAVEDEFCONFIG
1282
1283 def update_defconfig(self):
1284 """Update the input defconfig and go back to the idle state."""
1285
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001286 log = self.parser.check_defconfig()
1287 if log:
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001288 self.suspicious_boards.add(self.defconfig)
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001289 self.log += log
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001290 orig_defconfig = os.path.join('configs', self.defconfig)
1291 new_defconfig = os.path.join(self.build_dir, 'defconfig')
1292 updated = not filecmp.cmp(orig_defconfig, new_defconfig)
1293
1294 if updated:
Joe Hershberger06cc1d32016-06-10 14:53:30 -05001295 self.log += color_text(self.options.color, COLOR_LIGHT_BLUE,
Masahiro Yamadae307fa92016-06-08 11:47:37 +09001296 "defconfig was updated.\n")
1297
1298 if not self.options.dry_run and updated:
1299 shutil.move(new_defconfig, orig_defconfig)
1300 self.finish(True)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001301
Masahiro Yamada4efef992016-05-19 15:52:03 +09001302 def finish(self, success):
1303 """Display log along with progress and go to the idle state.
Masahiro Yamada1d085562016-05-19 15:52:02 +09001304
1305 Arguments:
Masahiro Yamada4efef992016-05-19 15:52:03 +09001306 success: Should be True when the defconfig was processed
1307 successfully, or False when it fails.
Masahiro Yamada1d085562016-05-19 15:52:02 +09001308 """
1309 # output at least 30 characters to hide the "* defconfigs out of *".
1310 log = self.defconfig.ljust(30) + '\n'
1311
1312 log += '\n'.join([ ' ' + s for s in self.log.split('\n') ])
1313 # Some threads are running in parallel.
1314 # Print log atomically to not mix up logs from different threads.
Masahiro Yamada4efef992016-05-19 15:52:03 +09001315 print >> (sys.stdout if success else sys.stderr), log
1316
1317 if not success:
1318 if self.options.exit_on_error:
1319 sys.exit("Exit on error.")
1320 # If --exit-on-error flag is not set, skip this board and continue.
1321 # Record the failed board.
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001322 self.failed_boards.add(self.defconfig)
Masahiro Yamada4efef992016-05-19 15:52:03 +09001323
Masahiro Yamada1d085562016-05-19 15:52:02 +09001324 self.progress.inc()
1325 self.progress.show()
Masahiro Yamada4efef992016-05-19 15:52:03 +09001326 self.state = STATE_IDLE
Masahiro Yamada1d085562016-05-19 15:52:02 +09001327
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001328 def get_failed_boards(self):
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001329 """Returns a set of failed boards (defconfigs) in this slot.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001330 """
1331 return self.failed_boards
1332
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001333 def get_suspicious_boards(self):
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001334 """Returns a set of boards (defconfigs) with possible misconversion.
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001335 """
Masahiro Yamada916224c2016-08-22 22:18:21 +09001336 return self.suspicious_boards - self.failed_boards
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001337
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001338class Slots:
1339
1340 """Controller of the array of subprocess slots."""
1341
Simon Glass6821a742017-07-10 14:47:47 -06001342 def __init__(self, toolchains, configs, options, progress,
1343 reference_src_dir, db_queue):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001344 """Create a new slots controller.
1345
1346 Arguments:
Simon Glass6821a742017-07-10 14:47:47 -06001347 toolchains: Toolchains object containing toolchains.
Masahiro Yamadab134bc12016-05-19 15:51:57 +09001348 configs: A list of CONFIGs to move.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001349 options: option flags.
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001350 progress: A progress indicator.
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001351 reference_src_dir: Determine the true starting config state from this
1352 source tree.
Simon Glassd73fcb12017-06-01 19:39:02 -06001353 db_queue: output queue to write config info for the database
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001354 """
1355 self.options = options
1356 self.slots = []
1357 devnull = get_devnull()
1358 make_cmd = get_make_cmd()
1359 for i in range(options.jobs):
Simon Glass6821a742017-07-10 14:47:47 -06001360 self.slots.append(Slot(toolchains, configs, options, progress,
1361 devnull, make_cmd, reference_src_dir,
1362 db_queue))
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001363
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001364 def add(self, defconfig):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001365 """Add a new subprocess if a vacant slot is found.
1366
1367 Arguments:
1368 defconfig: defconfig name to be put into.
1369
1370 Returns:
1371 Return True on success or False on failure
1372 """
1373 for slot in self.slots:
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001374 if slot.add(defconfig):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001375 return True
1376 return False
1377
1378 def available(self):
1379 """Check if there is a vacant slot.
1380
1381 Returns:
1382 Return True if at lease one vacant slot is found, False otherwise.
1383 """
1384 for slot in self.slots:
1385 if slot.poll():
1386 return True
1387 return False
1388
1389 def empty(self):
1390 """Check if all slots are vacant.
1391
1392 Returns:
1393 Return True if all the slots are vacant, False otherwise.
1394 """
1395 ret = True
1396 for slot in self.slots:
1397 if not slot.poll():
1398 ret = False
1399 return ret
1400
1401 def show_failed_boards(self):
1402 """Display all of the failed boards (defconfigs)."""
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001403 boards = set()
Masahiro Yamada96dccd92016-06-15 14:33:53 +09001404 output_file = 'moveconfig.failed'
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001405
1406 for slot in self.slots:
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001407 boards |= slot.get_failed_boards()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001408
Masahiro Yamada96dccd92016-06-15 14:33:53 +09001409 if boards:
1410 boards = '\n'.join(boards) + '\n'
1411 msg = "The following boards were not processed due to error:\n"
1412 msg += boards
1413 msg += "(the list has been saved in %s)\n" % output_file
1414 print >> sys.stderr, color_text(self.options.color, COLOR_LIGHT_RED,
1415 msg)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001416
Masahiro Yamada96dccd92016-06-15 14:33:53 +09001417 with open(output_file, 'w') as f:
1418 f.write(boards)
Joe Hershberger2559cd82015-05-19 13:21:22 -05001419
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001420 def show_suspicious_boards(self):
1421 """Display all boards (defconfigs) with possible misconversion."""
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001422 boards = set()
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001423 output_file = 'moveconfig.suspicious'
1424
1425 for slot in self.slots:
Masahiro Yamada09c6c062016-08-22 22:18:20 +09001426 boards |= slot.get_suspicious_boards()
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001427
1428 if boards:
1429 boards = '\n'.join(boards) + '\n'
1430 msg = "The following boards might have been converted incorrectly.\n"
1431 msg += "It is highly recommended to check them manually:\n"
1432 msg += boards
1433 msg += "(the list has been saved in %s)\n" % output_file
1434 print >> sys.stderr, color_text(self.options.color, COLOR_YELLOW,
1435 msg)
1436
1437 with open(output_file, 'w') as f:
1438 f.write(boards)
1439
Masahiro Yamada5cc42a52016-06-15 14:33:51 +09001440class ReferenceSource:
1441
1442 """Reference source against which original configs should be parsed."""
1443
1444 def __init__(self, commit):
1445 """Create a reference source directory based on a specified commit.
1446
1447 Arguments:
1448 commit: commit to git-clone
1449 """
1450 self.src_dir = tempfile.mkdtemp()
1451 print "Cloning git repo to a separate work directory..."
1452 subprocess.check_output(['git', 'clone', os.getcwd(), '.'],
1453 cwd=self.src_dir)
1454 print "Checkout '%s' to build the original autoconf.mk." % \
1455 subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip()
1456 subprocess.check_output(['git', 'checkout', commit],
1457 stderr=subprocess.STDOUT, cwd=self.src_dir)
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001458
1459 def __del__(self):
Masahiro Yamada5cc42a52016-06-15 14:33:51 +09001460 """Delete the reference source directory
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001461
1462 This function makes sure the temporary directory is cleaned away
1463 even if Python suddenly dies due to error. It should be done in here
1464 because it is guaranteed the destructor is always invoked when the
1465 instance of the class gets unreferenced.
1466 """
Masahiro Yamada5cc42a52016-06-15 14:33:51 +09001467 shutil.rmtree(self.src_dir)
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001468
Masahiro Yamada5cc42a52016-06-15 14:33:51 +09001469 def get_dir(self):
1470 """Return the absolute path to the reference source directory."""
1471
1472 return self.src_dir
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001473
Simon Glass6821a742017-07-10 14:47:47 -06001474def move_config(toolchains, configs, options, db_queue):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001475 """Move config options to defconfig files.
1476
1477 Arguments:
Masahiro Yamadab134bc12016-05-19 15:51:57 +09001478 configs: A list of CONFIGs to move.
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001479 options: option flags
1480 """
Masahiro Yamadab134bc12016-05-19 15:51:57 +09001481 if len(configs) == 0:
Masahiro Yamada6a9f79f2016-05-19 15:52:09 +09001482 if options.force_sync:
1483 print 'No CONFIG is specified. You are probably syncing defconfigs.',
Simon Glassd73fcb12017-06-01 19:39:02 -06001484 elif options.build_db:
1485 print 'Building %s database' % CONFIG_DATABASE
Masahiro Yamada6a9f79f2016-05-19 15:52:09 +09001486 else:
1487 print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.',
1488 else:
1489 print 'Move ' + ', '.join(configs),
1490 print '(jobs: %d)\n' % options.jobs
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001491
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001492 if options.git_ref:
Masahiro Yamada5cc42a52016-06-15 14:33:51 +09001493 reference_src = ReferenceSource(options.git_ref)
1494 reference_src_dir = reference_src.get_dir()
1495 else:
Masahiro Yamadaf432c332016-06-15 14:33:52 +09001496 reference_src_dir = None
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001497
Joe Hershberger91040e82015-05-19 13:21:19 -05001498 if options.defconfigs:
Masahiro Yamada0dbc9b52016-10-19 14:39:54 +09001499 defconfigs = get_matched_defconfigs(options.defconfigs)
Joe Hershberger91040e82015-05-19 13:21:19 -05001500 else:
Masahiro Yamada684c3062016-07-25 19:15:28 +09001501 defconfigs = get_all_defconfigs()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001502
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001503 progress = Progress(len(defconfigs))
Simon Glass6821a742017-07-10 14:47:47 -06001504 slots = Slots(toolchains, configs, options, progress, reference_src_dir,
1505 db_queue)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001506
1507 # Main loop to process defconfig files:
1508 # Add a new subprocess into a vacant slot.
1509 # Sleep if there is no available slot.
Masahiro Yamadac5e60fd2016-05-19 15:51:55 +09001510 for defconfig in defconfigs:
1511 while not slots.add(defconfig):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001512 while not slots.available():
1513 # No available slot: sleep for a while
1514 time.sleep(SLEEP_TIME)
1515
1516 # wait until all the subprocesses finish
1517 while not slots.empty():
1518 time.sleep(SLEEP_TIME)
1519
Joe Hershberger2e2ce6c2015-05-19 13:21:25 -05001520 print ''
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001521 slots.show_failed_boards()
Masahiro Yamadafc2661e2016-06-15 14:33:54 +09001522 slots.show_suspicious_boards()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001523
Simon Glasscb008832017-06-15 21:39:33 -06001524def find_kconfig_rules(kconf, config, imply_config):
1525 """Check whether a config has a 'select' or 'imply' keyword
1526
1527 Args:
Tom Rini65e05dd2019-09-20 17:42:09 -04001528 kconf: Kconfiglib.Kconfig object
Simon Glasscb008832017-06-15 21:39:33 -06001529 config: Name of config to check (without CONFIG_ prefix)
1530 imply_config: Implying config (without CONFIG_ prefix) which may or
1531 may not have an 'imply' for 'config')
1532
1533 Returns:
1534 Symbol object for 'config' if found, else None
1535 """
Tom Rini65e05dd2019-09-20 17:42:09 -04001536 sym = kconf.syms.get(imply_config)
Simon Glasscb008832017-06-15 21:39:33 -06001537 if sym:
Ulf Magnusson4e1102f2017-09-19 12:52:55 +02001538 for sel in sym.get_selected_symbols() | sym.get_implied_symbols():
Simon Glasscb008832017-06-15 21:39:33 -06001539 if sel.get_name() == config:
1540 return sym
1541 return None
1542
1543def check_imply_rule(kconf, config, imply_config):
1544 """Check if we can add an 'imply' option
1545
1546 This finds imply_config in the Kconfig and looks to see if it is possible
1547 to add an 'imply' for 'config' to that part of the Kconfig.
1548
1549 Args:
Tom Rini65e05dd2019-09-20 17:42:09 -04001550 kconf: Kconfiglib.Kconfig object
Simon Glasscb008832017-06-15 21:39:33 -06001551 config: Name of config to check (without CONFIG_ prefix)
1552 imply_config: Implying config (without CONFIG_ prefix) which may or
1553 may not have an 'imply' for 'config')
1554
1555 Returns:
1556 tuple:
1557 filename of Kconfig file containing imply_config, or None if none
1558 line number within the Kconfig file, or 0 if none
1559 message indicating the result
1560 """
Tom Rini65e05dd2019-09-20 17:42:09 -04001561 sym = kconf.syms.get(imply_config)
Simon Glasscb008832017-06-15 21:39:33 -06001562 if not sym:
1563 return 'cannot find sym'
1564 locs = sym.get_def_locations()
1565 if len(locs) != 1:
1566 return '%d locations' % len(locs)
1567 fname, linenum = locs[0]
1568 cwd = os.getcwd()
1569 if cwd and fname.startswith(cwd):
1570 fname = fname[len(cwd) + 1:]
1571 file_line = ' at %s:%d' % (fname, linenum)
1572 with open(fname) as fd:
1573 data = fd.read().splitlines()
1574 if data[linenum - 1] != 'config %s' % imply_config:
1575 return None, 0, 'bad sym format %s%s' % (data[linenum], file_line)
1576 return fname, linenum, 'adding%s' % file_line
1577
1578def add_imply_rule(config, fname, linenum):
1579 """Add a new 'imply' option to a Kconfig
1580
1581 Args:
1582 config: config option to add an imply for (without CONFIG_ prefix)
1583 fname: Kconfig filename to update
1584 linenum: Line number to place the 'imply' before
1585
1586 Returns:
1587 Message indicating the result
1588 """
1589 file_line = ' at %s:%d' % (fname, linenum)
1590 data = open(fname).read().splitlines()
1591 linenum -= 1
1592
1593 for offset, line in enumerate(data[linenum:]):
1594 if line.strip().startswith('help') or not line:
1595 data.insert(linenum + offset, '\timply %s' % config)
1596 with open(fname, 'w') as fd:
1597 fd.write('\n'.join(data) + '\n')
1598 return 'added%s' % file_line
1599
1600 return 'could not insert%s'
1601
1602(IMPLY_MIN_2, IMPLY_TARGET, IMPLY_CMD, IMPLY_NON_ARCH_BOARD) = (
1603 1, 2, 4, 8)
Simon Glass9b2a2e82017-06-15 21:39:32 -06001604
1605IMPLY_FLAGS = {
1606 'min2': [IMPLY_MIN_2, 'Show options which imply >2 boards (normally >5)'],
1607 'target': [IMPLY_TARGET, 'Allow CONFIG_TARGET_... options to imply'],
1608 'cmd': [IMPLY_CMD, 'Allow CONFIG_CMD_... to imply'],
Simon Glasscb008832017-06-15 21:39:33 -06001609 'non-arch-board': [
1610 IMPLY_NON_ARCH_BOARD,
1611 'Allow Kconfig options outside arch/ and /board/ to imply'],
Simon Glass9b2a2e82017-06-15 21:39:32 -06001612};
1613
Simon Glasscb008832017-06-15 21:39:33 -06001614def do_imply_config(config_list, add_imply, imply_flags, skip_added,
1615 check_kconfig=True, find_superset=False):
Simon Glass99b66602017-06-01 19:39:03 -06001616 """Find CONFIG options which imply those in the list
1617
1618 Some CONFIG options can be implied by others and this can help to reduce
1619 the size of the defconfig files. For example, CONFIG_X86 implies
1620 CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and
1621 all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to
1622 each of the x86 defconfig files.
1623
1624 This function uses the moveconfig database to find such options. It
1625 displays a list of things that could possibly imply those in the list.
1626 The algorithm ignores any that start with CONFIG_TARGET since these
1627 typically refer to only a few defconfigs (often one). It also does not
1628 display a config with less than 5 defconfigs.
1629
1630 The algorithm works using sets. For each target config in config_list:
1631 - Get the set 'defconfigs' which use that target config
1632 - For each config (from a list of all configs):
1633 - Get the set 'imply_defconfig' of defconfigs which use that config
1634 -
1635 - If imply_defconfigs contains anything not in defconfigs then
1636 this config does not imply the target config
1637
1638 Params:
1639 config_list: List of CONFIG options to check (each a string)
Simon Glasscb008832017-06-15 21:39:33 -06001640 add_imply: Automatically add an 'imply' for each config.
Simon Glass9b2a2e82017-06-15 21:39:32 -06001641 imply_flags: Flags which control which implying configs are allowed
1642 (IMPLY_...)
Simon Glasscb008832017-06-15 21:39:33 -06001643 skip_added: Don't show options which already have an imply added.
1644 check_kconfig: Check if implied symbols already have an 'imply' or
1645 'select' for the target config, and show this information if so.
Simon Glass99b66602017-06-01 19:39:03 -06001646 find_superset: True to look for configs which are a superset of those
1647 already found. So for example if CONFIG_EXYNOS5 implies an option,
1648 but CONFIG_EXYNOS covers a larger set of defconfigs and also
1649 implies that option, this will drop the former in favour of the
1650 latter. In practice this option has not proved very used.
1651
1652 Note the terminoloy:
1653 config - a CONFIG_XXX options (a string, e.g. 'CONFIG_CMD_EEPROM')
1654 defconfig - a defconfig file (a string, e.g. 'configs/snow_defconfig')
1655 """
Simon Glasscb008832017-06-15 21:39:33 -06001656 kconf = KconfigScanner().conf if check_kconfig else None
1657 if add_imply and add_imply != 'all':
1658 add_imply = add_imply.split()
1659
Simon Glass99b66602017-06-01 19:39:03 -06001660 # key is defconfig name, value is dict of (CONFIG_xxx, value)
1661 config_db = {}
1662
1663 # Holds a dict containing the set of defconfigs that contain each config
1664 # key is config, value is set of defconfigs using that config
1665 defconfig_db = collections.defaultdict(set)
1666
1667 # Set of all config options we have seen
1668 all_configs = set()
1669
1670 # Set of all defconfigs we have seen
1671 all_defconfigs = set()
1672
1673 # Read in the database
1674 configs = {}
1675 with open(CONFIG_DATABASE) as fd:
1676 for line in fd.readlines():
1677 line = line.rstrip()
1678 if not line: # Separator between defconfigs
1679 config_db[defconfig] = configs
1680 all_defconfigs.add(defconfig)
1681 configs = {}
1682 elif line[0] == ' ': # CONFIG line
1683 config, value = line.strip().split('=', 1)
1684 configs[config] = value
1685 defconfig_db[config].add(defconfig)
1686 all_configs.add(config)
1687 else: # New defconfig
1688 defconfig = line
1689
1690 # Work through each target config option in tern, independently
1691 for config in config_list:
1692 defconfigs = defconfig_db.get(config)
1693 if not defconfigs:
1694 print '%s not found in any defconfig' % config
1695 continue
1696
1697 # Get the set of defconfigs without this one (since a config cannot
1698 # imply itself)
1699 non_defconfigs = all_defconfigs - defconfigs
1700 num_defconfigs = len(defconfigs)
1701 print '%s found in %d/%d defconfigs' % (config, num_defconfigs,
1702 len(all_configs))
1703
1704 # This will hold the results: key=config, value=defconfigs containing it
1705 imply_configs = {}
1706 rest_configs = all_configs - set([config])
1707
1708 # Look at every possible config, except the target one
1709 for imply_config in rest_configs:
Simon Glass9b2a2e82017-06-15 21:39:32 -06001710 if 'ERRATUM' in imply_config:
Simon Glass99b66602017-06-01 19:39:03 -06001711 continue
Simon Glass9b2a2e82017-06-15 21:39:32 -06001712 if not (imply_flags & IMPLY_CMD):
1713 if 'CONFIG_CMD' in imply_config:
1714 continue
1715 if not (imply_flags & IMPLY_TARGET):
1716 if 'CONFIG_TARGET' in imply_config:
1717 continue
Simon Glass99b66602017-06-01 19:39:03 -06001718
1719 # Find set of defconfigs that have this config
1720 imply_defconfig = defconfig_db[imply_config]
1721
1722 # Get the intersection of this with defconfigs containing the
1723 # target config
1724 common_defconfigs = imply_defconfig & defconfigs
1725
1726 # Get the set of defconfigs containing this config which DO NOT
1727 # also contain the taret config. If this set is non-empty it means
1728 # that this config affects other defconfigs as well as (possibly)
1729 # the ones affected by the target config. This means it implies
1730 # things we don't want to imply.
1731 not_common_defconfigs = imply_defconfig & non_defconfigs
1732 if not_common_defconfigs:
1733 continue
1734
1735 # If there are common defconfigs, imply_config may be useful
1736 if common_defconfigs:
1737 skip = False
1738 if find_superset:
1739 for prev in imply_configs.keys():
1740 prev_count = len(imply_configs[prev])
1741 count = len(common_defconfigs)
1742 if (prev_count > count and
1743 (imply_configs[prev] & common_defconfigs ==
1744 common_defconfigs)):
1745 # skip imply_config because prev is a superset
1746 skip = True
1747 break
1748 elif count > prev_count:
1749 # delete prev because imply_config is a superset
1750 del imply_configs[prev]
1751 if not skip:
1752 imply_configs[imply_config] = common_defconfigs
1753
1754 # Now we have a dict imply_configs of configs which imply each config
1755 # The value of each dict item is the set of defconfigs containing that
1756 # config. Rank them so that we print the configs that imply the largest
1757 # number of defconfigs first.
Simon Glasscb008832017-06-15 21:39:33 -06001758 ranked_iconfigs = sorted(imply_configs,
Simon Glass99b66602017-06-01 19:39:03 -06001759 key=lambda k: len(imply_configs[k]), reverse=True)
Simon Glasscb008832017-06-15 21:39:33 -06001760 kconfig_info = ''
1761 cwd = os.getcwd()
1762 add_list = collections.defaultdict(list)
1763 for iconfig in ranked_iconfigs:
1764 num_common = len(imply_configs[iconfig])
Simon Glass99b66602017-06-01 19:39:03 -06001765
1766 # Don't bother if there are less than 5 defconfigs affected.
Simon Glass9b2a2e82017-06-15 21:39:32 -06001767 if num_common < (2 if imply_flags & IMPLY_MIN_2 else 5):
Simon Glass99b66602017-06-01 19:39:03 -06001768 continue
Simon Glasscb008832017-06-15 21:39:33 -06001769 missing = defconfigs - imply_configs[iconfig]
Simon Glass99b66602017-06-01 19:39:03 -06001770 missing_str = ', '.join(missing) if missing else 'all'
1771 missing_str = ''
Simon Glasscb008832017-06-15 21:39:33 -06001772 show = True
1773 if kconf:
1774 sym = find_kconfig_rules(kconf, config[CONFIG_LEN:],
1775 iconfig[CONFIG_LEN:])
1776 kconfig_info = ''
1777 if sym:
1778 locs = sym.get_def_locations()
1779 if len(locs) == 1:
1780 fname, linenum = locs[0]
1781 if cwd and fname.startswith(cwd):
1782 fname = fname[len(cwd) + 1:]
1783 kconfig_info = '%s:%d' % (fname, linenum)
1784 if skip_added:
1785 show = False
1786 else:
Tom Rini65e05dd2019-09-20 17:42:09 -04001787 sym = kconf.syms.get(iconfig[CONFIG_LEN:])
Simon Glasscb008832017-06-15 21:39:33 -06001788 fname = ''
1789 if sym:
1790 locs = sym.get_def_locations()
1791 if len(locs) == 1:
1792 fname, linenum = locs[0]
1793 if cwd and fname.startswith(cwd):
1794 fname = fname[len(cwd) + 1:]
1795 in_arch_board = not sym or (fname.startswith('arch') or
1796 fname.startswith('board'))
1797 if (not in_arch_board and
1798 not (imply_flags & IMPLY_NON_ARCH_BOARD)):
1799 continue
1800
1801 if add_imply and (add_imply == 'all' or
1802 iconfig in add_imply):
1803 fname, linenum, kconfig_info = (check_imply_rule(kconf,
1804 config[CONFIG_LEN:], iconfig[CONFIG_LEN:]))
1805 if fname:
1806 add_list[fname].append(linenum)
1807
1808 if show and kconfig_info != 'skip':
1809 print '%5d : %-30s%-25s %s' % (num_common, iconfig.ljust(30),
1810 kconfig_info, missing_str)
1811
1812 # Having collected a list of things to add, now we add them. We process
1813 # each file from the largest line number to the smallest so that
1814 # earlier additions do not affect our line numbers. E.g. if we added an
1815 # imply at line 20 it would change the position of each line after
1816 # that.
1817 for fname, linenums in add_list.iteritems():
1818 for linenum in sorted(linenums, reverse=True):
1819 add_imply_rule(config[CONFIG_LEN:], fname, linenum)
Simon Glass99b66602017-06-01 19:39:03 -06001820
1821
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001822def main():
1823 try:
1824 cpu_count = multiprocessing.cpu_count()
1825 except NotImplementedError:
1826 cpu_count = 1
1827
1828 parser = optparse.OptionParser()
1829 # Add options here
Simon Glasscb008832017-06-15 21:39:33 -06001830 parser.add_option('-a', '--add-imply', type='string', default='',
1831 help='comma-separated list of CONFIG options to add '
1832 "an 'imply' statement to for the CONFIG in -i")
1833 parser.add_option('-A', '--skip-added', action='store_true', default=False,
1834 help="don't show options which are already marked as "
1835 'implying others')
Simon Glassd73fcb12017-06-01 19:39:02 -06001836 parser.add_option('-b', '--build-db', action='store_true', default=False,
1837 help='build a CONFIG database')
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001838 parser.add_option('-c', '--color', action='store_true', default=False,
1839 help='display the log in color')
Simon Glass9ede2122016-09-12 23:18:21 -06001840 parser.add_option('-C', '--commit', action='store_true', default=False,
1841 help='Create a git commit for the operation')
Joe Hershberger91040e82015-05-19 13:21:19 -05001842 parser.add_option('-d', '--defconfigs', type='string',
Simon Glassee4e61b2017-06-01 19:38:59 -06001843 help='a file containing a list of defconfigs to move, '
1844 "one per line (for example 'snow_defconfig') "
1845 "or '-' to read from stdin")
Simon Glass99b66602017-06-01 19:39:03 -06001846 parser.add_option('-i', '--imply', action='store_true', default=False,
1847 help='find options which imply others')
Simon Glass9b2a2e82017-06-15 21:39:32 -06001848 parser.add_option('-I', '--imply-flags', type='string', default='',
1849 help="control the -i option ('help' for help")
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001850 parser.add_option('-n', '--dry-run', action='store_true', default=False,
1851 help='perform a trial run (show log with no changes)')
1852 parser.add_option('-e', '--exit-on-error', action='store_true',
1853 default=False,
1854 help='exit immediately on any error')
Masahiro Yamada8513dc02016-05-19 15:52:08 +09001855 parser.add_option('-s', '--force-sync', action='store_true', default=False,
1856 help='force sync by savedefconfig')
Masahiro Yamada07913d12016-08-22 22:18:22 +09001857 parser.add_option('-S', '--spl', action='store_true', default=False,
1858 help='parse config options defined for SPL build')
Joe Hershberger2144f882015-05-19 13:21:20 -05001859 parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
1860 action='store_true', default=False,
1861 help='only cleanup the headers')
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001862 parser.add_option('-j', '--jobs', type='int', default=cpu_count,
1863 help='the number of jobs to run simultaneously')
Joe Hershberger6b96c1a2016-06-10 14:53:32 -05001864 parser.add_option('-r', '--git-ref', type='string',
1865 help='the git ref to clone for building the autoconf.mk')
Simon Glass6b403df2016-09-12 23:18:20 -06001866 parser.add_option('-y', '--yes', action='store_true', default=False,
1867 help="respond 'yes' to any prompts")
Joe Hershberger95bf9c72015-05-19 13:21:24 -05001868 parser.add_option('-v', '--verbose', action='store_true', default=False,
1869 help='show any build errors as boards are built')
Masahiro Yamadab6ef3932016-05-19 15:51:58 +09001870 parser.usage += ' CONFIG ...'
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001871
Masahiro Yamadab6ef3932016-05-19 15:51:58 +09001872 (options, configs) = parser.parse_args()
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001873
Simon Glass99b66602017-06-01 19:39:03 -06001874 if len(configs) == 0 and not any((options.force_sync, options.build_db,
1875 options.imply)):
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001876 parser.print_usage()
1877 sys.exit(1)
1878
Masahiro Yamadab6ef3932016-05-19 15:51:58 +09001879 # prefix the option name with CONFIG_ if missing
1880 configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
1881 for config in configs ]
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001882
Joe Hershberger2144f882015-05-19 13:21:20 -05001883 check_top_directory()
1884
Simon Glass99b66602017-06-01 19:39:03 -06001885 if options.imply:
Simon Glass9b2a2e82017-06-15 21:39:32 -06001886 imply_flags = 0
Simon Glassdee36c72017-07-10 14:47:46 -06001887 if options.imply_flags == 'all':
1888 imply_flags = -1
1889
1890 elif options.imply_flags:
1891 for flag in options.imply_flags.split(','):
1892 bad = flag not in IMPLY_FLAGS
1893 if bad:
1894 print "Invalid flag '%s'" % flag
1895 if flag == 'help' or bad:
1896 print "Imply flags: (separate with ',')"
1897 for name, info in IMPLY_FLAGS.iteritems():
1898 print ' %-15s: %s' % (name, info[1])
1899 parser.print_usage()
1900 sys.exit(1)
1901 imply_flags |= IMPLY_FLAGS[flag][0]
Simon Glass9b2a2e82017-06-15 21:39:32 -06001902
Simon Glasscb008832017-06-15 21:39:33 -06001903 do_imply_config(configs, options.add_imply, imply_flags,
1904 options.skip_added)
Simon Glass99b66602017-06-01 19:39:03 -06001905 return
1906
Simon Glassd73fcb12017-06-01 19:39:02 -06001907 config_db = {}
1908 db_queue = Queue.Queue()
1909 t = DatabaseThread(config_db, db_queue)
1910 t.setDaemon(True)
1911 t.start()
1912
Joe Hershberger2144f882015-05-19 13:21:20 -05001913 if not options.cleanup_headers_only:
Masahiro Yamadaf7536f72016-07-25 19:15:23 +09001914 check_clean_directory()
Simon Glass6821a742017-07-10 14:47:47 -06001915 bsettings.Setup('')
1916 toolchains = toolchain.Toolchains()
1917 toolchains.GetSettings()
1918 toolchains.Scan(verbose=False)
1919 move_config(toolchains, configs, options, db_queue)
Simon Glassd73fcb12017-06-01 19:39:02 -06001920 db_queue.join()
Joe Hershberger2144f882015-05-19 13:21:20 -05001921
Masahiro Yamada6a9f79f2016-05-19 15:52:09 +09001922 if configs:
Masahiro Yamadae9ea1222016-07-25 19:15:26 +09001923 cleanup_headers(configs, options)
Masahiro Yamada9ab02962016-07-25 19:15:29 +09001924 cleanup_extra_options(configs, options)
Chris Packhamca438342017-05-02 21:30:47 +12001925 cleanup_whitelist(configs, options)
Chris Packhamf90df592017-05-02 21:30:48 +12001926 cleanup_readme(configs, options)
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001927
Simon Glass9ede2122016-09-12 23:18:21 -06001928 if options.commit:
1929 subprocess.call(['git', 'add', '-u'])
1930 if configs:
1931 msg = 'Convert %s %sto Kconfig' % (configs[0],
1932 'et al ' if len(configs) > 1 else '')
1933 msg += ('\n\nThis converts the following to Kconfig:\n %s\n' %
1934 '\n '.join(configs))
1935 else:
1936 msg = 'configs: Resync with savedefconfig'
1937 msg += '\n\nRsync all defconfig files using moveconfig.py'
1938 subprocess.call(['git', 'commit', '-s', '-m', msg])
1939
Simon Glassd73fcb12017-06-01 19:39:02 -06001940 if options.build_db:
1941 with open(CONFIG_DATABASE, 'w') as fd:
1942 for defconfig, configs in config_db.iteritems():
Simon Glassc79d18c2017-08-13 16:02:54 -06001943 fd.write('%s\n' % defconfig)
Simon Glassd73fcb12017-06-01 19:39:02 -06001944 for config in sorted(configs.keys()):
Simon Glassc79d18c2017-08-13 16:02:54 -06001945 fd.write(' %s=%s\n' % (config, configs[config]))
1946 fd.write('\n')
Simon Glassd73fcb12017-06-01 19:39:02 -06001947
Masahiro Yamada5a27c732015-05-20 11:36:07 +09001948if __name__ == '__main__':
1949 main()