Skip to content
This repository was archived by the owner on Jan 26, 2022. It is now read-only.
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 52 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,64 @@
[![Build Status](https://ci.bazel.io/buildStatus/icon?job=BUILD_file_generator)](https://ci.bazel.io/job/BUILD_file_generator)

# BUILD File Generator
# BUILD File Generator (BFG)

BUILD File Generator generates Bazel BUILD files for Java code.
BUILD File Generator, or BFG for short, is a tool for generating Bazel BUILD files from one's source code. Instead of manually going through your project and creating BUILD files and their appropriate build rules, you can rely on BFG to do it for you!
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel more is less here :) does it really contribute? as an engineer I personally prefer as little prose as possible :)

In this respect, laying out in bullets what the project does (even if it simplifies things) makes it easier for me to quickly scan the readme and see if the project is right for me.


1. It reads all `.java` files, and extracts the class dependency graph.
2. Computes the strongly connected components of the graph.
3. For each component, creates a `java_library` rule.
An example project can be found here. TODO

### Why is it useful?

Bazel promises fast and correct builds, _especially_ for incremental builds. Rather than rebuild your entire codebase, Bazel _only_ rebuilds what is necessary, the targets that you have changed. For the remainder of your code, it relies on cached versions.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm super late to the party, so feel free to reject.
We also have different writing styles (and I'm not a native English speaker anyway) so there's probably no "better" approach here.

I suggest the following as a replacement for this section -

"""
Having all sources in a single BUILD rule doesn't allow Bazel to parallelize and cache builds. In order to fully benefit from Bazel, one must write multiple BUILD rules and connect them.

This project automates writing granular BUILD rules that allow Bazel to parallelize and cache builds.

It's useful to quickly try out Bazel on your project, as well as to periodically optimize your build graph.
"""


There is one caveat to this fast incremental performance, though. It is dependent on the granularity of your targets. At one extreme, the least granular end, you can define a single glob for your entire project. This would be the quickest way to get your project using Bazel. However, it comes at the cost of eradicating any of Bazel's incremental build performance. On the other end, you can try to manually define the most granular targets possible, a single rule per file, collapsing any cyclic dependencies into the same target. This would ensure you make use of Bazel's blazingly fast incremental builds. However, it is extremely tedious, and somewhat annoying to maintain.

BFG automates this difficult tango.

## Usage

TODO
BFG is composed of two general components

1. Language specific parsers (LSP)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps "parser" instead of "LSP" thorough the doc?

2. The BFG binary

The LSPs read your source code and generate a class dependency graph in the form of a protobuf. To generate your BUILD files, you pass the generated protobuf into the BFG binary.

### Step 0: Installation Instructions

To install BFG...TODO
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cgrushko shall I open an issue for "how to install bfg"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it much more than cloning the repo and building two targets?
Perhaps build the *_deploy.jar targets so the output can be copied around.


### Step 1: Using LSPs to generate dependency graphs

TODO(bazel-devel): add explanation and valid example arguments.

```bash
bazel run //lang:JavaSourceFileParserCli <TODO>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cgrushko how do you actually run the JavaSourceFileParserCli? For example what is the command line args for for google-java-format?

I wasn't able to immediately figure out what the command line format was, or how the input proto should be formatted.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For google-java-format it should probably be

bazel run //lang/java/src/main/java/com/google/devtools/build/bfg:JavaSourceFileParserCli --roots=core/src/main/java,core/src/test/java $(find core/src/main/java/ core/src/test/java/ -name \*.java) > bfg.bin

It's a bit silly that we have to pass both roots and files, but it comes from Google where you don't want to read the entire java tree just because it's the root. We can perhaps improve this by scanning all of roots if the user doesn't provide source files, but it's for another day.

It should print usage information if run without flags (but I haven't tried it lately). Does it now?
By format, what do you mean?

```

This generates a protobuf at TODO.
TODO(bazel-devel): add clear protocol for how the protos are structured. Useful for other developers who may want to write LSPs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can link to the proto definition,


### Step 2: Generating BUILD files using BFG binary

TODO(bazel-devel): add explanation and valid example arguments.

```bash
bazel run //src:bfg <TODO>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

//src/main/java/com/google/devtools/build/bfg ?

```

### Supported Languages

We currently support Java projects. The next language on our roadmap is Scala.

## Development

### Contributing

We welcome contributions! We generally prefer contributions that either (1) add features by extending BFG to other languages, (2) fix existing bugs, or (3) present bugs through the use of example projects.

## Adding or updating third-party dependencies
### Dependency Management

We use a third-party tool called [bazel-deps](https://github.com/johnynek/bazel-deps)
to manage our dependencies. All of our dependencies are listed in
To manage our dependencies, we use a third-party tool called [bazel-deps](https://github.com/johnynek/bazel-deps). All of our dependencies are listed in
[`maven_deps.yaml`](maven_deps.yaml). bazel-deps provides tools to manage
adding and updating dependencies in that file, and generates the Bazel build
files for them in `thirdparty/jvm/`.
Expand Down
7 changes: 7 additions & 0 deletions lang/BUILD
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
licenses(["notice"]) # Apache 2.0

java_binary(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you prefer not using //lang/java/src/main/java/com/google/devtools/build/bfg:JavaSourceFileParserCli ?

name = "JavaSourceFileParserCli",
main_class = "com.google.devtools.build.bfg.JavaSourceFileParserCli",
runtime_deps = ["//lang/java/src/main/java/com/google/devtools/build/bfg:JavaSourceFileParser"],
)
17 changes: 16 additions & 1 deletion lang/README.md
Original file line number Diff line number Diff line change
@@ -1 +1,16 @@
A directory for language-specific parsers.
# Language Specific Parsers

This directory contains a set of language-specific parsers, that can be used to generate a class and file level dependency graph of your project. It outputs this dependency graph as a protobuff. For users of bfg, this dependency graph can be used to generate your BUILD files.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"that can be used to generate a class..." -> that are used to provide BFG with dependency information.

"protobuff" - we can link to the definition,

"bfg" -> "BFG" ?


We currently only support Java. However, we plan to add Scala support, and welcome contributions for other languages.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd drop this sentence, it doesn't add information - the user can see which directories exist there


### Usage

```bash

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

superfluous newlines?

bazel run //lang:JavaSourceFileParserCli
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you prefer not using //lang/java/src/main/java/com/google/devtools/build/bfg:JavaSourceFileParserCli ?


```

TODO(bazel-devel): what do the command line options mean??? There is a TODO in the Cli asking how the files will be obtained.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cgrushko similar question as before. How do you actually run the JavaSourceFileParserCli?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above. I'm not sure what the TODO means (we should figure out, or delete it :)


6 changes: 1 addition & 5 deletions lang/java/src/main/java/com/google/devtools/build/bfg/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,6 @@ java_library(
"//thirdparty/jvm/com/google/guava",
"//thirdparty/jvm/org/eclipse/jdt:org_eclipse_jdt_core",
],
visibility = ["//lang:__subpackages__"],
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cgrushko is this the appropriate visibility for the JavaSourceFileParser?

For context, I moved the JavaSourceFileParserCli binary to the top of the directory. This was so one simply needed to type

//src:JavaSourceFileParserCli

rather than the more tedious

//src/main/java/com/google/devtools/build/bfg:JavaSourceFileParserCli

The previous approach was annoyingly tedious.

As far as the visibility, I want to expose JavaSourceFileParser to just JavaSourceFileParserCli and its subpackages. However, that introduces a circular dependencies.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, you can't set visibility to a single rule.

How often do you expect users to build the thing?
When I built it it was from an IDE so didn't have to type it at all.
To run it, I had a symlink, but this arguably raises the bar for contributions.

We can add an alias() to lang/java/BUILD that points at the full path, but tbh this seems like dead weight.

)

java_binary(
name = "JavaSourceFileParserCli",
main_class = "com.google.devtools.build.bfg.JavaSourceFileParserCli",
runtime_deps = [":JavaSourceFileParser"],
)
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,9 @@
import protos.com.google.devtools.build.bfg.Bfg;

/**
* Entry point to BFG java source file parser. Given a list of source files, it prints a graph viz
* dot file representing the class level dependencies between these source files.
* Entry point to the BFG java source file parser. Given a list of source files,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's leave this to another PR?
same for other code changes.

* it obtains the class and file level dependencies between the files, generates a
* dependency graph, and outputs this graph as a protobuf.
*/
public class JavaSourceFileParserCli {

Expand Down Expand Up @@ -73,7 +74,8 @@ private void run(String[] args) throws Exception {
} catch (CmdLineException e) {
if (sourceFiles.isEmpty()) {
System.err.println("Must provide file names to parse.");
} else {
//TODO(bazel-team) print example usage of the class
} else {
System.err.println(e.getMessage());
}
e.getParser().printUsage(System.err);
Expand Down
13 changes: 13 additions & 0 deletions src/BUILD
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
licenses(["notice"]) # Apache 2.0

package(default_visibility = [
"//visibility:private",
])

java_binary(
name = "bfg",
main_class = "com.google.devtools.build.bfg.Bfg",
stamp = 1,
runtime_deps = ["//src/main/java/com/google/devtools/build/bfg:Bfg"],
)

10 changes: 3 additions & 7 deletions src/main/java/com/google/devtools/build/bfg/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -76,13 +76,9 @@ java_library(
"//thirdparty/jvm/com/google/guava",
"//thirdparty/jvm/com/google/re2j",
],
)

java_binary(
name = "bfg",
main_class = "com.google.devtools.build.bfg.Bfg",
stamp = 1,
runtime_deps = [":Bfg"],
visibility = [
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cgrushko similar visibility questions.

"//src:__subpackages__",
],
)

java_library(
Expand Down
1 change: 1 addition & 0 deletions src/main/java/com/google/devtools/build/bfg/Bfg.java
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,7 @@ private static Pattern compilePattern(CmdLineParser cmdLineParser, String patter
}
}

//TODO(bazel-devel): actually print what the intended usage is.
private static void explainUsageErrorAndExit(CmdLineParser cmdLineParser, String message) {
System.err.println(message);
cmdLineParser.printUsage(System.err);
Expand Down