HBASE-29838 Run Hadoop Check as a GitHub Action by ndimiduk · Pull Request #7651 · apache/hbase

ndimiduk · 2026-01-20T07:05:05Z

No description provided.

ndimiduk · 2026-01-20T14:09:02Z

Looks like we occupied the runner for 6h and then it was aborted.

Apache9 · 2026-01-20T16:27:28Z

https://infra.apache.org/github-actions-policy.html

The policy here does not say about the 6 hours timeout...

We can ask infra about the rules and the size of the github runners, our jenkins runners finished in "343m 23s", which was very close to 6 hours, so if the machine of the github runner is weaker, the build will be very easy to cost more than 6 hours...

ndimiduk · 2026-01-20T16:35:32Z

My action itself has timeout-minutes: 600 . i'll try bumping this up.

ndimiduk · 2026-01-20T16:38:31Z

or not. 6h is GH's hard limit, https://docs.github.com/en/actions/reference/limits

Apache9 · 2026-01-21T01:52:22Z

Then maybe we should try self hosted github runners? For self hosted runners the execution time limit is 5 days...

ndimiduk · 2026-01-23T07:07:12Z

Then maybe we should try self hosted github runners? For self hosted runners the execution time limit is 5 days...

Yes we should bring this back to our CI discussions with Infra. Maybe we can borrow from the pool of new Jenkins workers while we continue to build this out.

Yetus is supposed to provide smart, selective detection of module changes when it decided which tests to run. I think the new .github directory broke that for this run, so I've pushed a change to exclude it, maybe that will help.

I'm also going to see if I can manually parallelize the unit test runs -- maybe break out three separate checks for the three main unit test groups or something like that.

Apache9 · 2026-01-23T09:05:27Z

Then maybe we should try self hosted github runners? For self hosted runners the execution time limit is 5 days...

Yes we should bring this back to our CI discussions with Infra. Maybe we can borrow from the pool of new Jenkins workers while we continue to build this out.

Yetus is supposed to provide smart, selective detection of module changes when it decided which tests to run. I think the new .github directory broke that for this run, so I've pushed a change to exclude it, maybe that will help.

I'm also going to see if I can manually parallelize the unit test runs -- maybe break out three separate checks for the three main unit test groups or something like that.

Maybe we could split the UTs run as a seperated github check? Or even more, we could split the UTs run as two seperated check, one has -PrunDevTests(for small/medium tests) and one has -PrunLargeTests(for large tests).

ndimiduk · 2026-01-23T12:37:28Z

Maybe we could split the UTs run as a seperated github check? Or even more, we could split the UTs run as two seperated check, one has -PrunDevTests(for small/medium tests) and one has -PrunLargeTests(for large tests).

Yep, that's exactly my thinking as well. Landing these other cleanup issues and I'll be back.

ndimiduk · 2026-01-23T13:20:52Z

Okay this is better. Module selection chose only hbase-examples for running the unit tests.

-1 overall
  
  | Vote |      Subsystem |  Runtime   | Comment
  ============================================================================
  |   0  |        reexec  |   1m 13s   | Docker mode activated. 
  |  -0  |         yetus  |   0m  4s   | Unprocessed flag(s): --brief-report-file 
  |      |                |            | --spotbugs-strict-precheck
  |      |                |            | --author-ignore-list
  |      |                |            | --blanks-eol-ignore-file
  |      |                |            | --blanks-tabs-ignore-file
  |      |                |            | --quick-hadoopcheck
  +---------------------------------------------------------------------------
  |      |                |            | Prechecks 
  +---------------------------------------------------------------------------
  +---------------------------------------------------------------------------
  |      |                |            | master Compile Tests 
  +---------------------------------------------------------------------------
  |  +1  |    mvninstall  |   3m 25s   | master passed 
  |  +1  |       compile  |   0m 19s   | master passed 
  |  +1  |       javadoc  |   0m 17s   | master passed 
  |  +1  |    shadedjars  |   4m 47s   | branch has no errors when building our 
  |      |                |            | shaded downstream artifacts.
  +---------------------------------------------------------------------------
  |      |                |            | Patch Compile Tests 
  +---------------------------------------------------------------------------
  |  +1  |    mvninstall  |   3m  7s   | the patch passed 
  |  +1  |       compile  |   0m 18s   | the patch passed 
  |  +1  |         javac  |   0m 18s   | the patch passed 
  |  +1  |       javadoc  |   0m 15s   | the patch passed 
  |  +1  |    shadedjars  |   4m 51s   | patch has no errors when building our 
  |      |                |            | shaded downstream artifacts.
  +---------------------------------------------------------------------------
  |      |                |            | Other Tests 
  +---------------------------------------------------------------------------
  |  -1  |          unit  |   0m 24s   | hbase-examples in the patch failed. 
  |      |                |  19m 12s   |
  
  
  || Subsystem || Report/Notes ||
  ============================================================================
  |  unit  | /home/runner/work/hbase/hbase/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-examples.txt |
  | Docker | ClientAPI=1.48 ServerAPI=1.48 base: /home/runner/work/hbase/hbase/yetus-jdk17-hadoop3-check/output/Dockerfile |
  | GITHUB PR | https://github.com/apache/hbase/pull/7651 |
  | Optional Tests | javac javadoc unit compile shadedjars |
  | uname | Linux 9b6b1363e88a 6.11.0-1018-azure #18~24.04.1-Ubuntu SMP Sat Jun 28 04:46:03 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
  | Build tool | maven |
  | Personality | dev-support/hbase-personality.sh |
  | git revision | master / adfd0560c941109283928946bd671a5836604e46 |
  | Default Java | Eclipse Adoptium-17.0.11+9 |
  | Max. process+thread count | 72 (vs. ulimit of 30000) |
  | modules | C: hbase-examples U: hbase-examples |
  | versions | git=2.34.1 maven=3.9.8 |
  | Powered by | Apache Yetus 0.15.0 https://yetus.apache.org/ |

ndimiduk · 2026-01-23T15:51:15Z

Looking over out last successful nightly on master, the Large tests on hbase-server still took 7h. I'm looking for other ways to partition this up.

Apache9

Let's also separated runSmallTests and runMediumTests?

Apache9 · 2026-02-03T15:06:25Z

pom.xml

+            <groupId>org.apache.maven.plugins</groupId>
+            <artifactId>maven-surefire-plugin</artifactId>
+            <configuration>
+              <includes>


Will this be overridden by the command line parameter?

Okay so what I found is, when using the Category selector + the wave include list + the exclude list, all these work together as you would hope: only Large tests are run, which match the include regex, and entries found in the exclude list are indeed excluded.

However, if the yetus feature that adds -Dtest.include.pattern, i think it'll override the includes defined here. I'm not sure if that feature is used... anyway, I think that I prefer these wave definitions live here in the pom, rather than up in the GHA or personality script or whatever.

pom.xml

Apache9 · 2026-02-03T15:13:11Z

Seems the include for large wave does not work as expected...

ndimiduk · 2026-02-03T15:19:55Z

Yeah, pattern matching doesn't support character ranges, or something. Also I noticed that we have some test files named like TestFoo but also FooTest, so addressing that.

I did see your note about breaking up the devTests profile. I'll make that change also.

ndimiduk · 2026-02-03T17:18:36Z

I spot-checked the tests run by large waves 1 and 3 (2 is still running) and they look to contain only the expected tests. I think we're good here.

Any other comments @Apache9 ?

Apache9 · 2026-02-04T10:38:49Z

.github/workflows/yetus-jdk17-hadoop3-unit-check.yml

+          GITHUB_PASSWORD: ${{ secrets.GITHUB_TOKEN }}
+          GITHUB_USER: ${{ github.actor }}
+          PATCHDIR: "${{ github.workspace }}/yetus-jdk17-hadoop3-unit-check/output"
+          PLUGINS: "github,htmlout,maven,unit"


Do we need maven plugin here? We do not need mvninstall check?

I dug into this earlier, and the answer seems to be no, the unit plugin has enough to go on, so long as maven plugin is also present. The mvninstall was redundant work from the compile check.

I mean can we remove maven plugin here? The console output for the unit check showed that we did run mvninstall check...

I think we need the maven plugin in order for the unit plugin to be able to run maven to run the tests. From reading the yetus code, maven is the "build tool" that is used by the "module worker".

OK, then could we add mvninstall in the TESTS_FILTER value to skip mvninstall check?

Apache-HBase · 2026-02-05T10:27:11Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/19/console in case of problems.

Apache-HBase · 2026-02-05T10:27:50Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/19/console in case of problems.

Apache-HBase · 2026-02-05T11:32:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 46s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+0 🆗	yamllint	0m 0s		yamllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ master Compile Tests _
+1 💚	mvninstall	4m 53s		master passed
+1 💚	compile	13m 38s		master passed
+1 💚	spotless	1m 6s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	4m 40s		the patch passed
+1 💚	compile	13m 11s		the patch passed
+1 💚	javac	13m 11s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	pylint	0m 5s	/results-pylint.txt	The patch generated 1 new + 13 unchanged - 6 fixed = 14 total (was 19)
+1 💚	shellcheck	0m 1s		No new issues.
+1 💚	xmllint	0m 0s		No new issues.
+1 💚	hadoopcheck	15m 28s		Patch does not cause any errors with Hadoop 3.3.6 3.4.1.
+1 💚	spotless	1m 9s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 16s		The patch does not generate ASF License warnings.
		66m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/19/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7651
Optional Tests	dupname asflicense codespell detsecrets shellcheck shelldocs spotless pylint yamllint javac xmllint hadoopcheck compile
uname	Linux de3161c33942 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `2dcbbd0`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	188 (vs. ulimit of 30000)
modules	C: . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7651/19/console
versions	git=2.34.1 maven=3.9.8 pylint=2.15.5 shellcheck=0.8.0 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2026-02-05T12:20:48Z

dev-support/hbase-personality.sh

 function get_include_exclude_tests_arg
 {
-  local  __resultvar=$1
+  local __resultvar=$1


a bunch of the changes to this function are no longer needed, since i'm taking the simpler approach. let me wind these back.