Java login Kerberos authentication expiration problem

This problem is a bug in the lower version of hadoop and will be automatically solved after upgrading to hadoop3.0.0+.

question:

Recently, the CDH cluster added kerberos authentication, and it was discovered that more than 24 hours after the JavaWeb application was started, the kerberos certificate expired, causing the query to Hbase to fail.

When the Spark program connects to CDH, through the principal and keytab configuration methods, the credentials will be internally transferred to hdfs for use by the Executor and Driver. When the voucher ticket is about to expire, the voucher will be regenerated through Keytab.

spark2-submit \
 --master yarn \
 --deploy-mode cluster \
 --num-executors 3 \
 --driver-memory 4g \
 --executor-memory 4g \
 --executor-cores 2 \
 --conf spark.default.parallelism=10 \
 --conf spark.shuffle.file.buffe=1024k \
 --conf spark.executor.memoryOverhead=4096 \
 --class com.hypers.streaming.PanoramaMain \
 --conf "spark.executor.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails" \
 --principal xxxx \
 --keytab xxxx.keytab \
  xxxx.jar

However, the JavaWeb service we developed ourselves does not have this mechanism, which causes the Ticket to expire when the program is running. Finally, when querying Hbase, it prompts that there is no valid credentials and the query fails.

Source code debugging:

After local debugging and source code verification, it was found that the expiration time of the Ticket was specified as a fixed time every time it was started, and was not updated subsequently. The final expiration caused the application to fail to query Hbase.

The source code was recompiled, and logs related to Ticket credentials were added. Finally, it was confirmed that the problem was that the Ticket did not update the expiration date, causing the certification to expire. The printed log is as follows:

checkTGTAndReloginFromKeytab! shouldRenewImmediatelyForTests:false, Time.now():1606492834061, getRefreshTime(tgt):1606526141000, tgt:Ticket (hex) = 
0000: 61 82 01 40 30 82 01 3C   A0 03 02 01 05 A1 0D 1B  a..@0..<........
0010: 0B 45 53 53 45 4E 43 45   2E 43 4F 4D A2 20 30 1E  .ESSENCE.COM. 0.
0020: A0 03 02 01 02 A1 17 30   15 1B 06 6B 72 62 74 67  .......0...krbtg
0030: 74 1B 0B 45 53 53 45 4E   43 45 2E 43 4F 4D A3 82  t..ESSENCE.COM..
0040: 01 02 30 81 FF A0 03 02   01 12 A1 03 02 01 01 A2  ..0.............
0050: 81 F2 04 81 EF B4 0C DB   10 F7 5F 24 41 A8 91 70  .........._$A..p
0060: A0 76 13 DE 32 C2 29 82   D3 2D D0 50 34 E3 A9 B1  .v..2.)..-.P4...
0070: 4B B0 CF 55 CF 2E 83 8C   CB AE 05 2E 77 C5 14 6B  K..U........w..k
0080: 56 B0 63 DD 32 A7 C9 BA   DE 51 9B FF 06 C5 01 6F  V.c.2....Q.....o
0090: 16 01 AD E2 BB C4 6E C5   B3 BC 35 FE 0D AF 20 49  ......n...5... I
00A0: CF F7 4C 90 C7 7F A0 F5   A1 19 A2 FF 3D FF AB 67  ..L.........=..g
00B0: EF 91 FA C4 ED 59 C1 53   38 7A BD B2 FF FD FC 84  .....Y.S8z......
00C0: E5 16 DE 9F 08 62 49 65   57 86 57 6D 03 A2 96 83  .....bIeW.Wm....
00D0: F9 80 1E E6 F7 E4 4B 4F   9C 00 55 B9 4A A2 DA E2  ......KO..U.J...
00E0: F2 01 86 11 1C B5 B1 01   9A F6 29 75 C2 D1 80 8A  ..........)u....
00F0: 90 7F 35 C2 D0 A2 65 C3   9A 8B 9C 00 5E 20 EB 6C  ..5...e.....^ .l
0100: CF 1A 04 FC 20 8C 7B 4B   98 0A 0F 08 36 EC 94 7E  .... ..K....6...
0110: AF 71 4D A1 E1 DA 95 4A   50 5A A8 1B 39 4A F0 B6  .qM....JPZ..9J..
0120: 99 60 71 C7 E4 05 1A 54   0C 05 50 C5 42 B0 97 08  .`q....T..P.B...
0130: 29 5A 48 7E 8F A7 C9 BD   A6 9B 19 E4 A7 2A ED 48  )ZH..........*.H
0140: 8F 5F 1A D2                                        ._..

Client Principal = @xx.COM
Server Principal = xxxx/[email protected]
Session Key = EncryptionKey: keyType=18 keyBytes (hex dump)=
0000: A2 ED 15 B4 25 87 D1 B8   70 23 96 41 48 4B B6 79  ....%...p#.AHK.y
0010: 96 60 1A 1D EF D7 50 C0   31 C6 F5 63 8A 65 56 9E  .`....P.1..c.eV.


Forwardable Ticket true
Forwarded Ticket false
Proxiable Ticket false
Proxy Ticket false
Postdated Ticket false
Renewable Ticket false
Initial Ticket false
Auth Time = Fri Nov 27 14:03:41 CST 2020
Start Time = Fri Nov 27 14:03:41 CST 2020
End Time = Sat Nov 28 14:03:41 CST 2020
Renew Till = null
Client Addresses  Null 
Mainly the End Time and Renew Till fields, explained in the source code:
/**
 * Returns the start time for this ticket's validity period.
 *
 * @return the start time for this ticket's validity period
 *         or null if not set.
 */
public final java.util.Date getStartTime() {
    return (startTime == null) ? null : (Date)startTime.clone();
}

/**
 * Returns the expiration time for this ticket's validity period.
 *
 * @return the expiration time for this ticket's validity period.
 */
public final java.util.Date getEndTime() {
    return (endTime == null) ? null : (Date) endTime.clone();
}

/**
 * Returns the latest expiration time for this ticket, including all
 * renewals. This will return a null value for non-renewable tickets.
 *
 * @return the latest expiration time for this ticket.
 */
public final java.util.Date getRenewTill() {
    return (renewTill == null) ? null: (Date)renewTill.clone();
}

Solutions:

The final solution we found was that we need to regularly refresh (re-login) the credentials ourselves to make the credentials permanent, or specify a longer ticket_lifetime. The validity period of the credentials allows the application to delay the expiration as much as possible.

reference:HBase Kerberos connection renewal strategy – Stack Overflow

Final solution:

When connecting to Hbase and triggering the first login to Kerberos, start a scheduled task and call the checkTGTAndReloginFromKeytab() method every time to update the credentials to achieve the effect of scheduled updates.

conf.addResource("hbase-site-test.xml");
                            conf.addResource("core-site-test.xml");
                            conf.addResource("hdfs-site-test.xml");
                            try {
                                System.setProperty("java.security.krb5.conf", Constants.KRB5_CONF_PATH);
                                UserGroupInformation.setConfiguration(conf);
                                UserGroupInformation.loginUserFromKeytab(Constants.KEYTAB_USER, Constants.KEYTAB_PATH);
                                startCheckKeytabTgtAndReloginJob(); // Scheduled credential update task
                            } catch (IOException e) {
                                logger.error("Login failure", e);
                            }
/**
     * * Update credentials regularly
     */
    private static void startCheckKeytabTgtAndReloginJob() {
        //The voucher will be updated when the 10-minute cycle reaches a certain range from the expiration time.
        ThreadPool.updateConfigThread.scheduleWithFixedDelay(() -> {
            try {
                UserGroupInformation.getLoginUser().checkTGTAndReloginFromKeytab();
                logger.warn("get tgt:{}", UserGroupInformation.getLoginUser().getTGT());
                logger.warn("Check Kerberos Tgt And Relogin From Keytab Finish.");
            } catch (IOException e) {
                logger.error("Check Kerberos Tgt And Relogin From Keytab Error", e);
            }
        }, 0, 10, TimeUnit.MINUTES);
        logger.warn("Start Check Keytab TGT And Relogin Job Success.");
    }

Note: It can also be updated through the loginFromTicketCache() method, but it depends on the authentication cache on the server. Or directly call kinit -R to update (refer to the renewal method in the Hadoop source code).

// Refer to the renewal method in Hadoop source code
            try {
                String cmd = "kinit";
                Shell.execCommand(cmd, "-R");
                log.warn("exec kinit -R!");
            } catch (IOException e) {
               log.warn("relogin error.",e);
            }
            try {
                UserGroupInformation.getLoginUser().reloginFromTicketCache();
            } catch (IOException e) {
                log.warn("relogin error.",e);
            }

Observation conclusion:

The validity period of the Ticket has been updated. It has been running for a few days and is working normally. Continue to observe later.

Related Posts

[Alibaba Cloud] SMS service

The value taken out from Redis by redisTemplate.opsForValue().get(KEY) is null and the stored key is quoted.

[Distributed cache] Distributed cache-caching technology

SpringBoot unit testing-JUnit5

java.lang.StackOverflowError

SpringBoot integrates Jwt (detailed steps + illustrations)

[Yunxiang·People] Huawei Cloud AI senior expert Bai Xiaolong: How does AI release application productivity and move towards AI engineering?

Tutorial on quickly setting up a java back-end development environment with Docker (detailed graphic and text combination)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

*