Last updated on June 4th, 2019 at 02:00 am
This is last updated on 6/4/2019, with up to date instruction and ct_log_list files. (Download is at the bottom)
I’ve wrote a script to automatically generate list files and put it on my blog media.
This is a tutorial about parsing a list of certificate transparency log servers (that are in OpenSSL requested formats)
If you use OpenSSL v1+, you might realize that
Some information about those flags:
-ct Request and parse SCTs (also enables OCSP stapling) -noct Do not request or parse SCTs (default) -ctlogfile infile CT log list CONF file
Basically, that’s a Certificate Transparency log that (mostly) CA embedded into your certificate, which is required by Chrome to not throw an error when you visit a site that has a certificate issued by this CA.
When you run the program with
openssl s_client -ct the first time (and every time you don’t have a CT log configuration file), you would (normally) expect an error like this
12580:error:02001002:system library:fopen:No such file or directory:crypto\bio\bss_file.c:74:fopen('C:\Program Files\Common Files\SSL/ct_log_list.cnf','rb') 12580:error:2006D080:BIO routines:BIO_new_file:no such file:crypto\bio\bss_file.c:81: 12580:error:0E078072:configuration file routines:def_load:no such file:crypto\conf\conf_def.c:150: 12580:error:3207B06D:CT routines:CTLOG_STORE_load_file:log conf invalid:crypto\ct\ct_log.c:209: This is from a windows device.
Now, how to resolve this issue? (or, what is a ct_log_list.cnf file and how could i get one?)
If you dig on google.com, you would found those (somewhat unhelpful) lines: Link
# This file specifies the Certificate Transparency logs # that are to be trusted. # Google's list of logs can be found here: # www.certificate-transparency.org/known-logs # A Python program to convert the log list to OpenSSL's format can be # found here: # https://github.com/google/certificate-transparency/blob/master/python/utilities/log_list/print_log_list.py # Use the "--openssl_output" flag.
Some talent users.. (not me) might figure out how to do it…. or like me… stuck at one error forever.. and struggle to find the answer.
Let’s find a way to parse the log.. (If you don’t want to see the steps, scroll down to the bottom & I have one .cnf file parsed)
- have python,
wget, with command line cli(and could execute py files in commandline)
- familiar with command line
- could install dependencies by yourself
- Linux Machine (or please use the parsed ct_log_list.cnf on the end of this tutorial)
1. Go to the link.
You’ll need to see Google’s official repo before you download it… (make sure it’s still valid & exist)
Actually, go to this link https://github.com/google/certificate-transparency/tree/master/python/utilities/log_list since you would need to download all files from this folder.
2. Make a directory at a local machine (preferably Linux..) & download all ct tools from Google (Github repo)
Go to your machine, create one folder (name whatever you want)
Download all files in this folder
Commands to execute (Step 2)
cd ~/ #move to your home folder mkdir ct_logs #make a directory called ct_logs, under your userhome cd ~/ct_logs/ # Download all necessary python files from google ct tools wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/print_log_list.py wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/cpp_generator.py wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/java_generator.py wget https://raw.githubusercontent.com/google/certificate-transparency/master/python/utilities/log_list/openssl_generator.py
3. Download ct_lists from Google (certificate transparency)
First, visit http://www.certificate-transparency.org/known-logs, which is a list of all logs chrome currently have & added
When you run print_all_log.py, it’s going to ask for several files, else it would throw an error.
List of files needed:
- log_list.json (with flag –log_list=log_list.json) (This is the file that contains all log servers included in chrome) or all_logs_list.json (with flag –log_list=all_logs_list.json) (all ct servers announced)
- log_list_pubkey.pem(with flag –signer_key=log_list_pubkey.pem) (Signature file, log_list.json ‘s signer public key)
- log_list.sig (with flag –signature=log_list.sig) (Signature file, prove that log_list is not modified)
Now, we could download all three files from http://www.certificate-transparency.org/known-logs.
Download log_list.json with this url: https://www.gstatic.com/ct/log_list/log_list.json , log_list.sig from https://www.gstatic.com/ct/log_list/log_list.sig , log_list_pubkey.pem from https://www.gstatic.com/ct/log_list/log_list_pubkey.pem
Command to execute (Step 3)
cd ~/ct_logs/ # Download all necessary ct log & signature files from google wget https://www.gstatic.com/ct/log_list/log_list.json wget https://www.gstatic.com/ct/log_list/log_list.sig wget https://www.gstatic.com/ct/log_list/log_list_pubkey.pem wget https://www.gstatic.com/ct/log_list/all_logs_list.json wget https://www.gstatic.com/ct/log_list/log_list_schema.json
4. install all dependencies
Now you could run the python file
cd ~/ct_logs/ python print_log_list.py
You might see this error output:
Traceback (most recent call last): File "print_log_list.py", line 11, in <module> from absl import flags as gflags ImportError: No module named absl
pip install absl-py
Now run the script again, you would see another error
Traceback (most recent call last): File "print_log_list.py", line 12, in <module> import jsonschema ImportError: No module named jsonschema
This means jsonschema package is missing, you would install it from pip
sudo pip install jsonschema
Run the script, yet another error
Traceback (most recent call last): File "print_log_list.py", line 13, in <module> import M2Crypto ImportError: No module named M2Crypto
Install m2crypto from pip
sudo pip install m2crypto
After this, the script should be running, (but throw error like the following ones)
Traceback (most recent call last): File "print_log_list.py", line 115, in <module> sys.argv = FLAGS(sys.argv) File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 633, in __call__ self._assert_all_validators() File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 507, in _assert_all_validators self._assert_validators(all_validators) File "/home/pi/.local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 528, in _assert_validators raise _exceptions.IllegalFlagValueError('%s: %s' % (message, str(e))) absl.flags._exceptions.IllegalFlagValueError: flag --log_list=None: Flag --log_list must be specified.
This means you are on the path, proceed to the next step
5. parse the log file
This would require a few flags with files, we’ve downloaded all in the previous steps.
Now, if you want to use the CT log server that
python print_log_list.py --log_list_schema log_list_schema.json --log_list log_list.json --signer_key log_list_pubkey.pem --signature log_list.sig --openssl_output ooo
Now, if you want to use all announced CT log server, run the below command, replace ‘
python print_log_list.py --log_list all_logs_list.json --signer_key log_list_pubkey.pem --signature log_list.sig --openssl_output ooo
If no errors are present, here you go..
You are done, open the file & place it to where your openssl wants
The below files are OpenSSL ct-logs (Generated on 4/18/2019)
The “openssl-certificate-transparency-log-included” is a list of CT Logs that are currently compliant with Chrome’s CT policy (or have been and were disqualified), and are included in Chrome.
The “openssl-certificate-transparency-log-full” is a list of all known and announced CT Logs.
No matter what file you downloaded, you’ll need to rename it to either ct_log_list.cnf or ct_log_list.conf, depend on what the openssl wants.
Have questions or feedback? Leave it in comment area below